{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 《强化学习：原理与Python实现》更新与勘误\n",
    "\n",
    "（2019年8月第1版第1次印刷）\n",
    "\n",
    "### 行数计算方法\n",
    "\n",
    "本勘误文档中，行数计算“第$i$行”（$i=0,1,2,\\ldots$）是从0开始计数的。小节标题、公式、内联代码、注意、本章要点记入行数，章标题、图、表、代码清单及它们的题注不计入行数。空行不计入行数。\n",
    "\n",
    "$\n",
    "\\newcommand{\\sfA}{\\unicode{x1d608}}\n",
    "\\newcommand{\\sfR}{\\unicode{x1d619}}\n",
    "\\newcommand{\\sfS}{\\unicode{x1d61a}}\n",
    "\\newcommand{\\sfa}{\\unicode{x1d622}}\n",
    "\\newcommand{\\sfs}{\\unicode{x1d634}}\n",
    "\\newcommand{\\sfx}{\\unicode{x1d639}}\n",
    "\\newcommand{\\bftheta}{\\pmb{\\unicode{x3B8}}}\n",
    "\\newcommand{\\eat}{\\unicode{x5403}}\n",
    "\\newcommand{\\noeat}{\\unicode{x4E0D}\\unicode{x5403}}\n",
    "\\newcommand{\\full}{\\unicode{x9971}}\n",
    "\\newcommand{\\hungry}{\\unicode{x997F}}\n",
    "\\newcommand{\\E}{\\textrm{E}}\n",
    "$\n",
    "\n",
    "## 第5页第19行\n",
    "\n",
    "记为$\\sfR$\n",
    "\n",
    "### 改为\n",
    "\n",
    "记为$R$\n",
    "\n",
    "## 第11页第17行\n",
    "\n",
    "动作空间是Dicrete(3)\n",
    "\n",
    "### 改为\n",
    "\n",
    "动作空间是Discrete(3)\n",
    "\n",
    "## 第20页第2行\n",
    "\n",
    "作者注：\n",
    "这种带折扣的回报定义既可以用于回合制任务，也可以用于连续性任务，是一种统一的表示。\n",
    "\n",
    "\n",
    "## 第20页第9行\n",
    "\n",
    "为$\\bar{R}=\\lim\\limits_{t\\to+\\infty}\\E\\left[\\frac{1}{t}\\sum\\limits_{\\tau=0}^{t}R_\\tau\\right]$\n",
    "\n",
    "### 改为\n",
    "\n",
    "为$\\bar{R}=\\lim\\limits_{t\\to+\\infty}\\E\\left[\\frac{1}{t}\\sum\\limits_{\\tau=1}^{t}R_\\tau\\right]$\n",
    "\n",
    "\n",
    "## 第20页第24-27行\n",
    "\n",
    "$q_\\pi\\left(\\eat\\mid\\hungry\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\hungry,\\sfA_t=\\eat\\right]$\n",
    "\n",
    "$q_\\pi\\left(\\noeat\\mid\\hungry\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\hungry,\\sfA_t=\\noeat\\right]$\n",
    "\n",
    "$q_\\pi\\left(\\eat\\mid\\full\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\full,\\sfA_t=\\eat\\right]$\n",
    "\n",
    "$q_\\pi\\left(\\noeat\\mid\\full\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\full,\\sfA_t=\\noeat\\right]$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$q_\\pi\\left(\\hungry,\\eat\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\hungry,\\sfA_t=\\eat\\right]$\n",
    "\n",
    "$q_\\pi\\left(\\hungry,\\noeat\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\hungry,\\sfA_t=\\noeat\\right]$\n",
    "\n",
    "$q_\\pi\\left(\\full,\\eat\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\full,\\sfA_t=\\eat\\right]$\n",
    "\n",
    "$q_\\pi\\left(\\full,\\noeat\\right)=\\E_\\pi\\left[G_t\\mid\\sfS_t=\\full,\\sfA_t=\\noeat\\right]$\n",
    "\n",
    "## 第2.2节和第2.3节\n",
    "\n",
    "有观点认为：用动作价值表示状态价值的表达式，或是用状态价值表示动作价值的表达式，不能称为Bellman方程；而用动作价值表示动作价值的表达式，或是用状态价值表示状态价值的表达式，才可以称为Bellman方程。\n",
    "\n",
    "## 第22页第18行\n",
    "\n",
    "${q_\\pi}\\left(\\sfs,\\sfa\\right)=\\gamma \\sum\\limits_{\\sfs',r}{p\\left(\\sfs',r\\mid\\sfs,\\sfa\\right)\\left[r+\\sum\\limits_{\\sfa'}{\\pi\\left(\\sfa'|\\sfs'\\right){q_\\pi}\\left(\\sfs',\\sfa'\\right)}\\right]}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "${q_\\pi}\\left(\\sfs,\\sfa\\right)=\\sum\\limits_{\\sfs',r}{p\\left(\\sfs',r\\mid\\sfs,\\sfa\\right)\\left[r+\\gamma \\sum\\limits_{\\sfa'}{\\pi\\left(\\sfa'|\\sfs'\\right){q_\\pi}\\left(\\sfs',\\sfa'\\right)}\\right]}$\n",
    "\n",
    "## 第23页第19行\n",
    "\n",
    "$\\begin{bmatrix}1&0&x-1&-x&0&0\\\\\\\\0&1&0&0&-y&y-1\\\\\\\\-\\gamma&0&1&0&0&0\\\\\\\\\\left(\\alpha-1\\right)\\gamma&-\\alpha\\gamma&0&1&0&0\\\\\\\\-\\beta\\gamma&\\left(\\beta-1\\right)\\gamma&0&0&1&0\\\\\\\\0&-\\gamma&0&0&0&1\\end{bmatrix}\\begin{bmatrix}v_\\pi\\left(\\hungry\\right)\\\\\\\\v_\\pi\\left(\\full\\right)\\\\\\\\q_\\pi\\left(\\hungry,\\noeat\\right)\\\\\\\\q_\\pi\\left(\\hungry,\\eat\\right)\\\\\\\\q_\\pi\\left(\\full,\\noeat\\right)\\\\\\\\q_\\pi\\left(\\full,\\eat\\right)\\end{bmatrix}=\\begin{bmatrix}0\\\\\\\\0\\\\\\\\2\\\\\\\\-4\\alpha+3\\\\\\\\4\\beta-2\\\\\\\\-1\\end{bmatrix}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\begin{bmatrix}1&0&x-1&-x&0&0\\\\\\\\0&1&0&0&-y&y-1\\\\\\\\-\\gamma&0&1&0&0&0\\\\\\\\\\left(\\alpha-1\\right)\\gamma&-\\alpha\\gamma&0&1&0&0\\\\\\\\-\\beta\\gamma&\\left(\\beta-1\\right)\\gamma&0&0&1&0\\\\\\\\0&-\\gamma&0&0&0&1\\end{bmatrix}\\begin{bmatrix}v_\\pi\\left(\\hungry\\right)\\\\\\\\v_\\pi\\left(\\full\\right)\\\\\\\\q_\\pi\\left(\\hungry,\\noeat\\right)\\\\\\\\q_\\pi\\left(\\hungry,\\eat\\right)\\\\\\\\q_\\pi\\left(\\full,\\noeat\\right)\\\\\\\\q_\\pi\\left(\\full,\\eat\\right)\\end{bmatrix}=\\begin{bmatrix}0\\\\\\\\0\\\\\\\\-2\\\\\\\\4\\alpha-3\\\\\\\\-4\\beta+2\\\\\\\\1\\end{bmatrix}$\n",
    "\n",
    "## 第24页代码清单2-1\n",
    "\n",
    "### 代码改为\n",
    "\n",
    "```\n",
    "import sympy\n",
    "from sympy import symbols\n",
    "sympy.init_printing()\n",
    "v_hungry, v_full = symbols('v_hungry v_full')\n",
    "q_hungry_eat, q_hungry_none, q_full_eat, q_full_none = \\\n",
    "        symbols('q_hungry_eat q_hungry_none q_full_eat q_full_none')\n",
    "alpha, beta, x, y, gamma = symbols('alpha beta x y gamma')\n",
    "system = sympy.Matrix((\n",
    "        (1, 0, x-1, -x, 0, 0, 0),\n",
    "        (0, 1, 0, 0, -y, y-1, 0),\n",
    "        (-gamma, 0, 1, 0, 0, 0, -2),\n",
    "        ((alpha-1)*gamma, -alpha*gamma, 0, 1, 0, 0, 4*alpha-3),\n",
    "        (-beta*gamma, (beta-1)*gamma, 0, 0, 1, 0, -4*beta+2),\n",
    "        (0, -gamma, 0, 0, 0, 1, 1) )) # 标准形式的系数矩阵\n",
    "sympy.solve_linear_system(system,\n",
    "        v_hungry, v_full,\n",
    "        q_hungry_none, q_hungry_eat, q_full_none, q_full_eat)\n",
    "```\n",
    "\n",
    "## 第24页第2行~第11行\n",
    "\n",
    "### 求解有误，更改如下\n",
    "\n",
    "$v_\\pi\\left(\\hungry\\right)=\\frac{1}{\\Delta}\\left(\\alpha\\gamma xy-3\\alpha\\gamma x+4\\alpha x-\\beta\\gamma xy-2\\beta\\gamma y+\\gamma x+2\\gamma-x-2\\right)$\n",
    "\n",
    "$v_\\pi\\left(\\full\\right)=\\frac{1}{\\Delta}\\left(\\alpha\\gamma xy+\\alpha\\gamma x-\\beta\\gamma xy+2\\beta\\gamma y-4\\beta y-\\gamma y-\\gamma+y+1\\right)$\n",
    "\n",
    "$q_\\pi\\left(\\hungry,\\noeat\\right)=\\frac{1}{\\Delta}\\left(\\alpha\\gamma^2xy-\\alpha\\gamma^2x+2\\alpha\\gamma x-\\beta\\gamma^2xy-2\\beta\\gamma y+\\gamma^2x-\\gamma x+2\\gamma-2\\right)$\n",
    "\n",
    "$q_\\pi\\left(\\hungry,\\eat\\right)=\\frac{1}{\\Delta}\\left(\\alpha\\gamma^2xy-\\alpha\\gamma^2x-\\alpha\\gamma^2y+\\alpha\\gamma^2+2\\alpha\\gamma x+\\alpha\\gamma y-5\\alpha\\gamma+4\\alpha-\\beta\\gamma^2xy+\\beta\\gamma^2y-\\right.$\n",
    "\n",
    "$\\left.3\\beta\\gamma y+\\gamma^2x-\\gamma^2-\\gamma x+4\\gamma-3\\right)$\n",
    "\n",
    "$q_\\pi\\left(\\full,\\noeat\\right)=\\frac{1}{\\Delta}\\left(\\alpha\\gamma^2xy-\\alpha\\gamma^2x+2\\alpha\\gamma x-\\beta\\gamma^2xy+\\beta\\gamma^2x+\\beta\\gamma^2y-\\beta\\gamma^2-\\beta\\gamma x-3\\beta\\gamma y+\\right.$\n",
    "\n",
    "$\\left.5\\beta\\gamma-4\\beta-\\gamma^2{y}+\\gamma^2+\\gamma{y}-3\\gamma+2\\right)$\n",
    "\n",
    "$q_\\pi\\left(\\full,\\eat\\right)=\\frac{1}{\\Delta}\\left(\\alpha\\gamma^2xy+\\alpha\\gamma x-\\beta\\gamma^2xy+\\beta\\gamma^2y-3\\beta\\gamma y-\\gamma^2y+\\gamma y-\\gamma+1\\right)$\n",
    "\n",
    "其中\n",
    "\n",
    "$\\Delta=\\left(1-\\gamma\\right)\\left(1-\\left(1-\\alpha{x}-\\beta{y}\\right)\\gamma\\right)$\n",
    "\n",
    "## 第25页第2行\n",
    "\n",
    "如果对于任意$\\sfs\\in\\mathcal{S}$，都${v_{\\pi}}\\left(\\sfs\\right)\\le{v_*}\\left(\\sfs\\right)$，则称\n",
    "\n",
    "### 改为\n",
    "\n",
    "如果对于任意$\\sfs\\in\\mathcal{S}$都满足${v_{\\pi}}\\left(\\sfs\\right)\\le{v_*}\\left(\\sfs\\right)$，则称\n",
    "\n",
    "## 第26页第2行和第7行\n",
    "\n",
    "$\\gamma\\sum_{\\sfs'}{p\\left(\\sfs',r\\mid\\sfs,\\sfa\\right)}v_*\\left(\\sfs'\\right)$\n",
    "\n",
    "（共2处）\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\gamma\\sum_{\\sfs'}{p\\left(\\sfs'\\mid\\sfs,\\sfa\\right)}v_*\\left(\\sfs'\\right)$\n",
    "\n",
    "## 第27页第0~5行\n",
    "\n",
    "所有的下标$\\pi$改为下标$\\ast$（共16处）。\n",
    "\n",
    "\n",
    "## 第27页代码清单2-2\n",
    "\n",
    "### 代码改为\n",
    "\n",
    "```\n",
    "import sympy\n",
    "from sympy import symbols\n",
    "sympy.init_printing()\n",
    "alpha, beta, gamma = symbols('alpha beta gamma')\n",
    "v_hungry, v_full = symbols('v_hungry v_full')\n",
    "q_hungry_eat, q_hungry_none, q_full_eat, q_full_none = \\\n",
    "        symbols('q_hungry_eat q_hungry_none q_full_eat q_full_none')\n",
    "xy_tuples = ((0, 0), (1, 0), (0, 1), (1, 1))\n",
    "for x, y in xy_tuples: # 分类讨论\n",
    "    system = sympy.Matrix((\n",
    "            (1, 0, x-1, -x, 0, 0, 0),\n",
    "            (0, 1, 0, 0, -y, y-1, 0),\n",
    "            (-gamma, 0, 1, 0, 0, 0, -2),\n",
    "            ((alpha-1)*gamma, -alpha*gamma, 0, 1, 0, 0, 4*alpha-3),\n",
    "            (-beta*gamma, (beta-1)*gamma, 0, 0, 1, 0, -4*beta+2),\n",
    "            (0, -gamma, 0, 0, 0, 1, 1) ))\n",
    "    result = sympy.solve_linear_system(system,\n",
    "            v_hungry, v_full,\n",
    "            q_hungry_none, q_hungry_eat, q_full_none, q_full_eat, simplification=True)\n",
    "    msgx = 'v(饿) = q(饿,{}吃)'.format('' if x else '不')\n",
    "    msgy = 'v(饱) = q(饱,{}吃)'.format('不' if y else '')\n",
    "    print('==== {}, {} ==== x = {}, y = {} ===='.format(msgx, msgy, x, y))\n",
    "    display(result)\n",
    "```\n",
    "\n",
    "## 第27页第10行~第29页第8行\n",
    "\n",
    "### 求解有误，更改如下\n",
    "\n",
    "接下来进一步分析这个方程组。比较最优价值的方程组和一般策略的价值方程组可以发现，最优策略价值方程组的解正是在一般策略方程组的解中$\\left(x,y\\right)$分别取(0,0)、(0,1)、(1,0)、(1,1)得到。所以，我们用一般策略价值方程组的解的形式来表示最优价值方程组的解，并比较其中$q_\\ast\\left(\\right.\\full$,$\\noeat\\left.\\right)$和$q_\\ast\\left(\\full,\\eat\\right)$的大小，以及$q_\\ast\\left(\\hungry,\\noeat\\right)$和$q_\\ast\\left(\\hungry,\\eat\\right)$的大小。在比较时，注意到$x,y\\in\\left[0,1\\right]$并且已经假设$\\alpha,\\beta,\\gamma\\in\\left(0,1\\right)$。所以可以验证\n",
    "\n",
    "$\\Delta=\\left(1-\\gamma\\right)\\left(1-\\left(1-\\alpha{x}-\\beta{y}\\right)\\gamma\\right)>0$\n",
    "\n",
    "进而比较大小可以只比较分母部分。\n",
    "\n",
    "首先来比较$q_\\ast\\left(\\hungry,\\noeat\\right)$和$q_\\ast\\left(\\hungry,\\eat\\right)$的大小。注意到$q_\\ast\\left(\\hungry,\\noeat\\right)\\le{q_\\ast}\\left(\\hungry,\\noeat\\right)$等价于\n",
    "\n",
    "$\\alpha\\gamma^2xy-\\alpha\\gamma^2x+2\\alpha\\gamma x-\\beta\\gamma^2xy-2\\beta\\gamma y+\\gamma^2x-\\gamma x+2\\gamma-2\\le$\n",
    "\n",
    "$\\alpha\\gamma^2xy-\\alpha\\gamma^2x-\\alpha\\gamma^2y+\\alpha\\gamma^2+2\\alpha\\gamma x+\\alpha\\gamma y-5\\alpha\\gamma+4\\alpha-\\beta\\gamma^2xy+\\beta\\gamma^2y-3\\beta\\gamma y+\\gamma^2x-\\gamma^2-\\gamma x+4\\gamma-3$\n",
    "\n",
    "即\n",
    "\n",
    "$-\\alpha\\gamma^2y+\\alpha\\gamma^2+\\alpha\\gamma y-5\\alpha\\gamma+4\\alpha+\\beta\\gamma^2y-\\beta\\gamma y-\\gamma^2+2\\gamma-1\\ge0$\n",
    "\n",
    "两边除以$\\left(1-\\gamma\\right)$可得\n",
    "\n",
    "$\\alpha\\gamma{y}-\\alpha\\gamma+4\\alpha-\\beta\\gamma y+\\gamma-1\\ge0$\n",
    "\n",
    "注意到$1-\\alpha+\\left(\\alpha-\\beta\\right)y>0$，上述不等式等价于\n",
    "\n",
    "$\\gamma\\ge\\frac{1-4\\alpha}{1-\\alpha+\\left(\\alpha-\\beta\\right)y}$.\n",
    "\n",
    "接下来比较$q_\\ast\\left(\\full,\\noeat\\right)$和$q_\\ast\\left(\\full,\\eat\\right)$的大小。注意到$q_\\ast\\left(\\full,\\noeat\\right)\\le{q_\\ast}\\left(\\full,\\eat\\right)$等价于\n",
    "$\\alpha{\\gamma^2}xy-\\alpha{\\gamma^2}x+2\\alpha\\gamma{x}-\\beta{\\gamma^2}xy+\\beta{\\gamma^2}x+\\beta{\\gamma^2}y-\\beta{\\gamma^2}-\\beta\\gamma{x}-3\\beta\\gamma{y}+5\\beta\\gamma-4\\beta-{\\gamma^2}y+\\gamma^2+\\gamma{y}-3\\gamma+2\\le$\n",
    "\n",
    "$\\alpha{\\gamma^2}xy+\\alpha\\gamma{x}-\\beta{\\gamma^2}xy+\\beta{\\gamma^2}y-3\\beta\\gamma{y}-{\\gamma^2}y+\\gamma{y}-\\gamma+1$\n",
    "\n",
    "即\n",
    "\n",
    "$\\alpha{\\gamma^2}x-\\alpha\\gamma{x}-\\beta{\\gamma^2}x+\\beta\\gamma{x}+\\beta\\gamma^2-5\\beta\\gamma+4\\beta-\\gamma^2+2\\gamma-1\\ge{0}$\n",
    "\n",
    "两边除以$\\left(1-\\gamma\\right)$可得\n",
    "\n",
    "$-\\alpha\\gamma{x}+\\beta\\gamma{x}-\\beta\\gamma+4\\beta+\\gamma-1\\ge{0}$\n",
    "\n",
    "注意到$1-\\beta+\\left(\\beta-\\alpha\\right)x>0$，上述不等式等价于\n",
    "\n",
    "$\\gamma\\ge\\frac{1-4\\beta}{1-\\beta+\\left(\\beta-\\alpha\\right)x}$.\n",
    "\n",
    "综合上述分析，有以下四种情况。\n",
    "\n",
    "**情况I**：$q_\\ast\\left(\\hungry,\\noeat\\right)>{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)\\le{q_\\ast}\\left(\\full,\\eat\\right)$。这时有$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\noeat\\right)$且$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\eat\\right)$，以及$x=0$且$y=0$。相应条件简化为$\\gamma<\\frac{1-4\\alpha}{1-\\alpha}$且$\\gamma\\ge\\frac{1-4\\beta}{1-\\beta}$，最优价值简化为\n",
    "\n",
    "$q_\\ast\\left(\\hungry,\\eat\\right)=\\frac{1}{1-\\gamma}\\left(-\\alpha\\gamma+4\\alpha+\\gamma-3\\right)$\n",
    "\n",
    "$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\noeat\\right)=\\frac{-2}{1-\\gamma}$\n",
    "\n",
    "$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\eat\\right)=\\frac{1}{1-\\gamma}$\n",
    "\n",
    "$q_\\ast\\left(\\full,\\noeat\\right)=\\frac{1}{1-\\gamma}\\left(\\beta\\gamma+4\\beta-\\gamma+2\\right)$\n",
    "\n",
    "**情况II**：$q_\\ast\\left(\\hungry,\\noeat\\right)\\le{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)\\le{q_\\ast}\\left(\\full,\\eat\\right)$。这时有$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\eat\\right)$且$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\eat\\right)$，以及$x=1$且$y=0$。相应条件简化为$\\gamma\\ge\\frac{1-4\\alpha}{1-\\alpha}$且$\\gamma\\ge\\frac{1-4\\beta}{1-\\alpha}$，最优价值简化为\n",
    "\n",
    "$q_\\ast\\left(\\hungry,\\noeat\\right)=\\frac{1}{\\Delta_\\mathrm{II}}\\left(-\\alpha\\gamma^2+2\\alpha\\gamma+\\gamma^2+\\gamma-2\\right)$\n",
    "\n",
    "$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\eat\\right)=\\frac{1}{\\Delta_\\mathrm{II}}\\left(-3\\alpha\\gamma+4\\alpha+3\\gamma-3\\right)$\n",
    "\n",
    "$q_\\ast\\left(\\full,\\noeat\\right)=\\frac{1}{\\Delta_\\mathrm{II}}\\left(-\\alpha\\gamma^2+2\\alpha\\gamma+4\\beta\\gamma-4\\beta+\\gamma^2-3\\gamma+2\\right)$\n",
    "\n",
    "$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\eat\\right)=\\frac{1}{1-\\gamma}$\n",
    "\n",
    "其中\n",
    "\n",
    "$\\Delta_\\mathrm{II}=\\left(1-\\gamma\\right)\\left(1-\\left(1-\\alpha\\right)\\gamma\\right)$.\n",
    "\n",
    "**情况III**：$q_\\ast\\left(\\hungry,\\noeat\\right)>{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)>{q_\\ast}\\left(\\full,\\eat\\right)$。这时有$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\noeat\\right)$且$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\noeat\\right)$，以及$x=0$且$y=1$。相应条件简化为$\\gamma<\\frac{1-4\\alpha}{1-\\beta}$且$\\gamma<\\frac{1-4\\beta}{1-\\beta}$，最优价值简化为\n",
    "\n",
    "$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\noeat\\right)=\\frac{-2}{1-\\gamma}$\n",
    "\n",
    "$q_\\ast\\left(\\hungry,\\eat\\right)=\\frac{1}{\\Delta_\\mathrm{III}}\\left(-4\\alpha\\gamma+4\\alpha+\\beta\\gamma^2-3\\beta\\gamma-\\gamma^2+4\\gamma-3\\right)$\n",
    "\n",
    "$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\noeat\\right)=\\frac{1}{\\Delta_\\mathrm{III}}\\left(2\\beta\\gamma-4\\beta-2\\gamma+2\\right)$\n",
    "\n",
    "$q_\\ast\\left(\\full,\\eat\\right)=\\frac{1}{\\Delta_\\mathrm{III}}\\left(\\beta\\gamma^2-3\\beta\\gamma-\\gamma^2+1\\right)$\n",
    "\n",
    "其中\n",
    "\n",
    "$\\Delta_\\mathrm{III}=\\left(1-\\gamma\\right)\\left(1-\\left(1-\\beta\\right)\\gamma\\right)$.\n",
    "\n",
    "**情况IV**：$q_\\ast\\left(\\hungry,\\noeat\\right)\\le{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)>{q_\\ast}\\left(\\full,\\eat\\right)$。这时有$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\eat\\right)$且$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\noeat\\right)$，以及$x=1$且$y=1$。相应条件简化为$\\gamma\\ge\\frac{1-4\\alpha}{1-\\beta}$且$\\gamma<\\frac{1-4\\beta}{1-\\alpha}$，最优价值简化为\n",
    "\n",
    "$q_\\ast\\left(\\hungry,\\noeat\\right)=\\frac{1}{\\Delta_\\mathrm{IV}}{\\left(2\\alpha\\gamma-\\beta\\gamma^2-2\\beta\\gamma+\\gamma^2+\\gamma-2\\right)}$\n",
    "\n",
    "$v_\\ast\\left(\\hungry\\right)=q_\\ast\\left(\\hungry,\\eat\\right)=\\frac{1}{\\Delta_\\mathrm{IV}}{\\left(-2\\alpha\\gamma+4\\alpha-3\\beta\\gamma+3\\gamma-3\\right)}$\n",
    "\n",
    "$v_\\ast\\left(\\full\\right)=q_\\ast\\left(\\full,\\noeat\\right)=\\frac{1}{\\Delta_\\mathrm{IV}}{\\left(2\\alpha\\gamma+\\beta\\gamma-4\\beta-2\\gamma+2\\right)}$。\n",
    "\n",
    "$q_\\ast\\left(\\full,\\eat\\right)=\\frac{1}{\\Delta_\\mathrm{IV}}{\\left(\\alpha\\gamma^2+\\alpha\\gamma-3\\beta\\gamma-\\gamma^2+1\\right)}$\n",
    "\n",
    "其中\n",
    "\n",
    "$\\Delta_\\mathrm{IV}=\\left(1-\\gamma\\right)\\left(1-\\left(1-\\alpha-\\beta\\right)\\gamma\\right)$.\n",
    "\n",
    "\n",
    "## 第29页第13行\n",
    "\n",
    "$v\\left(\\sfs\\right)\\ge r\\left(\\sfs,\\sfa\\right)+\\gamma\\sum\\limits_{\\sfs'} {p\\left(\\sfs'\\mid\\sfs,\\sfa\\right){v_*}\\left(\\sfs'\\right)},\\quad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$v\\left(\\sfs\\right)\\ge r\\left(\\sfs,\\sfa\\right)+\\gamma\\sum\\limits_{\\sfs'}{p\\left(\\sfs'\\mid\\sfs,\\sfa \\right){v}\\left(\\sfs'\\right)},\\quad\\sfs\\in\\mathcal{S},\\sfa\\in\\mathcal{A}$\n",
    "\n",
    "作者注：用线性规划求解最优状态价值的详细证明可见《Markov Decision Processes: Discrete Stochastic Dynamic Programming》（Martin Puterman）的第6.9节。\n",
    "\n",
    "## 第29页第21行\n",
    "\n",
    "$v_\\pi\\left(\\hungry\\right)=10$，$v_\\pi\\left(\\full\\right)=8\\frac{3}{4}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$v_\\ast\\left(\\hungry\\right)=\\frac{35}{11}$，$v_\\ast\\left(\\full\\right)=5$\n",
    "\n",
    "## 第29页第23行\n",
    "\n",
    "$q_\\pi\\left(\\hungry,\\noeat\\right)=7\\frac{2}{3}$，$q_\\pi\\left(\\hungry,\\eat\\right)=10$，$q_\\pi\\left(\\full,\\noeat\\right)=6$，$q_\\pi\\left(\\full,\\eat\\right)=8\\frac{3}{4}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$q_\\ast\\left(\\hungry,\\noeat\\right)=\\frac{6}{11}$，$q_\\ast\\left(\\hungry,\\eat\\right)=\\frac{35}{11}$，$q_\\ast\\left(\\full,\\noeat\\right)=\\frac{21}{11}$，$q_\\ast\\left(\\full,\\eat\\right)=5$\n",
    "\n",
    "## 第30页第4行~第25行\n",
    "\n",
    "### 求解有误，更改如下\n",
    "\n",
    "**情况I**：$q_\\ast\\left(\\hungry,\\noeat\\right)>{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)\\le{q_\\ast}\\left(\\full,\\eat\\right)$，即$\\gamma<\\frac{1-4\\alpha}{1-\\alpha}$且$\\gamma\\ge\\frac{1-4\\beta}{1-\\beta}$。这种情况的最优策略是\n",
    "\n",
    "$\\pi_\\ast\\left(\\hungry\\right)=\\noeat$，$\\pi_\\ast\\left(\\full\\right)=\\eat$.\n",
    "\n",
    "即饿时不吃，饱时吃。\n",
    "\n",
    "**情况II**：$q_\\ast\\left(\\hungry,\\noeat\\right)\\le{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)\\le{q_\\ast}\\left(\\full,\\eat\\right)$，即$\\gamma\\ge\\frac{1-4\\alpha}{1-\\alpha}$且$\\gamma\\ge\\frac{1-4\\beta}{1-\\alpha}$。这种情况的最优策略是\n",
    "\n",
    "$\\pi_\\ast\\left(\\hungry\\right)=\\eat$，$\\pi_\\ast\\left(\\full\\right)=\\eat$.\n",
    "\n",
    "即一直吃。\n",
    "\n",
    "**情况III**：$q_\\ast\\left(\\hungry,\\noeat\\right)>{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)>{q_\\ast}\\left(\\full,\\eat\\right)$，即$\\gamma<\\frac{1-4\\alpha}{1-\\beta}$且$\\gamma<\\frac{1-4\\beta}{1-\\beta}$。这种情况的最优策略是\n",
    "\n",
    "$\\pi_\\ast\\left(\\hungry\\right)=\\noeat$，$\\pi_\\ast\\left(\\full\\right)=\\noeat$.\n",
    "\n",
    "即一直$\\noeat$。\n",
    "\n",
    "**情况IV**：$q_\\ast\\left(\\hungry,\\noeat\\right)\\le{q_\\ast}\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)>{q_\\ast}\\left(\\full,\\eat\\right)$，即$\\gamma\\ge\\frac{1-4\\alpha}{1-\\beta}$且$\\gamma<\\frac{1-4\\beta}{1-\\alpha}$。这种情况的最优策略是\n",
    "\n",
    "$\\pi_\\ast\\left(\\hungry\\right)=\\eat$，$\\pi_\\ast\\left(\\full\\right)=\\noeat$.\n",
    "\n",
    "即$\\hungry$时$\\eat$，$\\full$时$\\noeat$。\n",
    "\n",
    "对于一组特定的数值，求解则更加直接。例如，当$\\alpha=\\frac{2}{3}$，$\\beta=\\frac{3}{4}$，$\\gamma=\\frac{4}{5}$时，2.3.2节已经求得了最优动作价值，此时动作价值满足$q_\\ast\\left(\\hungry,\\noeat\\right)<q_\\ast\\left(\\hungry,\\eat\\right)$且$q_\\ast\\left(\\full,\\noeat\\right)<q_\\ast\\left(\\full,\\eat\\right)$。所以，它对应的最优策略为$\\pi_\\ast\\left(\\hungry\\right)=\\pi_\\ast\\left(\\full\\right)=\\eat$.\n",
    "\n",
    "## 第31页第12行\n",
    "\n",
    "回合奖励为各步奖励之。\n",
    "\n",
    "### 改为\n",
    "\n",
    "回合奖励为各步奖励之和。\n",
    "\n",
    "## 第32页代码清单2-4\n",
    "\n",
    "### 代码改为\n",
    "\n",
    "```\n",
    "def play_once(env, policy):\n",
    "    total_reward = 0\n",
    "    state = env.reset()\n",
    "    while True:\n",
    "        loc = np.unravel_index(state, env.shape)\n",
    "        print('状态 = {}, 位置 = {}'.format(state, loc), end=' ')\n",
    "        action = np.random.choice(env.nA, p=policy[state])\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        print('动作 = {}, 奖励 = {}'.format(action, reward))\n",
    "        total_reward += reward\n",
    "        if done:\n",
    "            break\n",
    "    return total_reward\n",
    "```\n",
    "\n",
    "## 第32页第10行\n",
    "\n",
    "Bellmn期望方程为\n",
    "\n",
    "### 改为\n",
    "\n",
    "Bellman期望方程为\n",
    "\n",
    "## 第33页第0行\n",
    "\n",
    "Bellan期望方程：\n",
    "\n",
    "### 改为\n",
    "\n",
    "Bellman期望方程：\n",
    "\n",
    "## 第33页代码清单2-6\n",
    "\n",
    "### 题注改为\n",
    "\n",
    "用Bellman方程求解状态价值和动作价值\n",
    "\n",
    "### 代码改为\n",
    "\n",
    "```\n",
    "def evaluate_bellman(env, policy, gamma=1.):\n",
    "    a, b = np.eye(env.nS), np.zeros((env.nS))\n",
    "    for state in range(env.nS - 1):\n",
    "        for action in range(env.nA):\n",
    "            pi = policy[state][action]\n",
    "            for p, next_state, reward, done in env.P[state][action]:\n",
    "                a[state, next_state] -= (pi * gamma * p)\n",
    "                b[state] += (pi * reward * p)\n",
    "    v = np.linalg.solve(a, b)\n",
    "    q = np.zeros((env.nS, env.nA))\n",
    "    for state in range(env.nS - 1):\n",
    "        for action in range(env.nA):\n",
    "            for p, next_state, reward, done in env.P[state][action]:\n",
    "                q[state][action] += ((reward + gamma * v[next_state]) * p)\n",
    "    return v, q\n",
    "```\n",
    "\n",
    "## 第34页第2行\n",
    "\n",
    "$v\\left(\\sfs\\right)-\\gamma\\sum\\limits_{\\sfs',r}{p\\left(\\sfs',r|\\sfs,\\sfa\\right)v_*\\left(\\sfs'\\right)}\\ge\\sum\\limits_{\\sfs',r}{rp\\left(\\sfs',r|\\sfs,\\sfa\\right)}\\qquad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$v\\left(\\sfs\\right)-\\gamma\\sum\\limits_{\\sfs',r}{p\\left(\\sfs',r|\\sfs,\\sfa\\right)v\\left(\\sfs'\\right)}\\ge\\sum\\limits_{\\sfs',r}{rp\\left(\\sfs',r|\\sfs,\\sfa\\right)}\\qquad\\sfs\\in\\mathcal{S},\\sfa\\in\\mathcal{A}$\n",
    "\n",
    "## 第35页第25行\n",
    "\n",
    "可以应用下列线性规划求解最优动作价值\n",
    "\n",
    "### 改为\n",
    "\n",
    "可以应用下列线性规划求解最优状态价值\n",
    "\n",
    "## 第36页第2行\n",
    "\n",
    "$v\\left(\\sfs\\right)\\ge r\\left(\\sfs,\\sfa\\right)+\\gamma\\sum\\limits_{\\sfs'}{p\\left(\\sfs'|\\sfs,\\sfa\\right){v_*}\\left(\\sfs'\\right)} ,\\quad \\sfs\\in{\\cal S}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$v\\left(\\sfs\\right)\\ge r\\left(\\sfs,\\sfa\\right)+\\gamma \\sum\\limits_{\\sfs'}{p\\left(\\sfs'|\\sfs,\\sfa\\right){v}\\left(\\sfs'\\right)},\\quad\\sfs\\in\\mathcal{S},\\sfa\\in \\mathcal{A}$\n",
    "\n",
    "## 第38页第11行和第14行\n",
    "\n",
    "这里的$d$都是指$d_\\infty$（共3处）。\n",
    "\n",
    "## 第38页第21行\n",
    "\n",
    "$d\\left(t\\left(\\sfx'\\right),t\\left(\\sfx''\\right)\\right)<d\\left(\\sfx',\\sfx''\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$d\\left(t\\left(\\sfx'\\right),t\\left(\\sfx''\\right)\\right)<\\gamma d\\left(\\sfx',\\sfx''\\right)$\n",
    "\n",
    "## 第38页第24~25行\n",
    "\n",
    "用动作价值表示动作价值的形式\n",
    "\n",
    "### 改为\n",
    "\n",
    "用状态价值表示状态价值的形式\n",
    "\n",
    "## 第39页第5行~第7行\n",
    "\n",
    "$\\left|t_\\pi\\left(v'\\right)\\left(\\sfs\\right)-t_\\pi\\left(v''\\right)\\left(\\sfs\\right)\\right|\\le\\gamma\\sum\\limits_\\sfa{\\pi\\left(\\sfa|\\sfs\\right)\\sum\\limits_{\\sfs'}{p\\left(\\sfs'|\\sfs,\\sfa\\right)}}\\mathop{\\max}\\limits_{\\sfs'} \\left|v'\\left(\\sfs'\\right)-v''\\left(\\sfs'\\right)\\right|=\\gamma d\\left(v',v''\\right)$\n",
    "\n",
    "考虑到$v',v''$是任取的，所以有\n",
    "\n",
    "$d\\left(t_\\pi\\left(v'\\right),t_\\pi\\left(v''\\right)\\right)\\le\\gamma d\\left(v',v''\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\left|t_\\pi\\left(v'\\right)\\left(\\sfs\\right)-t_\\pi\\left(v''\\right)\\left(\\sfs\\right)\\right|\\le\\gamma\\sum\\limits_\\sfa{\\pi\\left(\\sfa|\\sfs\\right)\\sum\\limits_{\\sfs'}{p\\left(\\sfs'|\\sfs,\\sfa\\right)}}\\mathop{\\max}\\limits_{\\sfs'}\\left|v'\\left(\\sfs'\\right)-v''\\left(\\sfs'\\right)\\right|=\\gamma d_\\infty\\left(v',v''\\right)$\n",
    "\n",
    "考虑到$\\sfs$是任取的，所以有\n",
    "\n",
    "$d_\\infty\\left(t_\\pi\\left(v'\\right),t_\\pi\\left(v''\\right)\\right)\\le\\gamma d_\\infty\\left(v',v''\\right)$\n",
    "\n",
    "## 第39页第12行~第13行\n",
    "\n",
    "$\\le\\max_\\sfa\\left|f'\\left(\\sfa'\\right)-f''\\left(\\sfa'\\right)\\right|$\n",
    "\n",
    "### 改为（共2处）\n",
    "\n",
    "$\\le\\max_\\sfa\\left|f'\\left(\\sfa\\right)-f''\\left(\\sfa\\right)\\right|$\n",
    "\n",
    "## 第39页第15行~第19行\n",
    "\n",
    "${t_*}\\left(v'\\right)\\left(\\sfs\\right)-{t_*}\\left(v''\\right)\\left(\\sfs\\right)$\n",
    "\n",
    "$=\\mathop{\\max}\\limits_{\\sfa\\in\\mathcal{A}}\\left[r\\left(\\sfs,\\sfa\\right)+\\gamma\\sum\\limits_{\\sfs'\\in{\\mathcal{S}}}{p\\left(\\sfs'\\mid\\sfs,\\sfa\\right)v'\\left(\\sfs'\\right)}\\right]-\\mathop{\\max}\\limits_{\\sfa\\in{\\mathcal{A}}}\\left[r\\left(\\sfs,\\sfa\\right)+\\gamma\\sum\\limits_{\\sfs'\\in{\\mathcal{S}}}p\\left(\\sfs'\\mid\\sfs,\\sfa\\right)v''\\left(\\sfs'\\right)\\right]$\n",
    "\n",
    "$\\le\\mathop{\\max}\\limits_{\\sfa\\in\\mathcal{A}}\\left|\\gamma\\sum\\limits_{\\sfs'\\in{\\mathcal{S}}}{p\\left(\\sfs'\\mid\\sfs,\\sfa\\right)\\left(v'\\left(\\sfs'\\right)-v''\\left(\\sfs'\\right)\\right)}\\right|$\n",
    "\n",
    "$\\le\\gamma\\left|v'\\left(\\sfs'\\right)-v''\\left(\\sfs'\\right)\\right|,$\n",
    "\n",
    "进而易知$\\left|t_*\\left(v'\\right)\\left(\\sfs\\right)-{t_*}\\left(v''\\right)\\left(\\sfs\\right)\\right| \\le\\gamma\\left|v'\\left(\\sfs'\\right)- v''\\left(\\sfs'\\right)\\right|\\le\\gamma d\\left(v',v''\\right)$，所以$t_*$是压缩映射。\n",
    "\n",
    "### 改为\n",
    "\n",
    "${t_*}\\left(v'\\right)\\left(\\sfs\\right)-{t_*}\\left(v''\\right)\\left(\\sfs\\right)$\n",
    "\n",
    "$=\\mathop{\\max}\\limits_{\\sfa\\in\\mathcal{A}}\\left[r\\left(\\sfs,\\sfa\\right)+\\gamma\\sum\\limits_{\\sfs'\\in\\mathcal{S}}{p\\left(\\sfs'\\mid\\sfs,\\sfa\\right)v'\\left(\\sfs'\\right)}\\right]-\\mathop{\\max}\\limits_{\\sfa\\in\\mathcal{A}}\\left[r\\left(\\sfs,\\sfa\\right)+\\gamma\\sum\\limits_{\\sfs'\\in\\mathcal{S}}p\\left(\\sfs'\\mid\\sfs,\\sfa\\right)v''\\left(\\sfs'\\right)\\right]$\n",
    "\n",
    "$\\le\\mathop{\\max}\\limits_{\\sfa\\in\\mathcal{A}}\\left|{\\gamma\\sum\\limits_{\\sfs'\\in{\\mathcal{S}}}{p\\left(\\sfs'\\mid\\sfs,\\sfa\\right)\\left(v'\\left(\\sfs'\\right)-v''\\left(\\sfs'\\right)\\right)}}\\right|$\n",
    "\n",
    "$\\le\\gamma\\mathop{\\max}\\limits_{\\sfa\\in\\mathcal{A}}\\left|{\\sum\\limits_{\\sfs'\\in\\mathcal{S}}{p\\left(\\sfs'|\\sfs,\\sfa\\right)}}\\right|\\mathop{\\max}\\limits_{\\sfs'\\in\\mathcal{S}}\\left|v'\\left(\\sfs'\\right)-v''\\left(\\sfs'\\right)\\right|$\n",
    "\n",
    "$\\le\\gamma{d}_\\infty\\left(v',v''\\right)$\n",
    "\n",
    "进而易知$\\left|{{t_*}\\left(v'\\right)\\left(\\sfs\\right)-{t_*}\\left(v''\\right)\\left(\\sfs\\right)}\\right|\\le\\gamma d_\\infty\\left(v',v''\\right)$。所以$t_*$是压缩映射。\n",
    "\n",
    "\n",
    "## 第39页第21行\n",
    "\n",
    "如果$x\\in\\mathcal{X}$使得$t\\left(x\\right)=x$，则称$x$是\n",
    "\n",
    "### 改为\n",
    "\n",
    "如果$\\sfx\\in\\mathcal{X}$使得$t\\left(\\sfx\\right)=\\sfx$，则称$\\sfx$是\n",
    "\n",
    "\n",
    "## 第42页第7行\n",
    "\n",
    "对于两个确定性的策略\n",
    "\n",
    "### 改为\n",
    "\n",
    "对于策略\n",
    "\n",
    "\n",
    "## 第42页第8行\n",
    "\n",
    "$v_\\pi\\left(\\sfs\\right)\\le{q}_\\pi\\left(\\sfs,\\pi'\\left( \\sfs\\right)\\right),\\quad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$v_\\pi\\left(\\sfs\\right)\\le\\sum\\limits_\\sfa{\\pi'\\left(\\sfa|\\sfs\\right)q_\\pi\\left(\\sfs,\\sfa\\right)},\\quad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "\n",
    "## 第42页第12行~第43页第11行\n",
    "\n",
    "### 证明更新如下\n",
    "\n",
    "不等式（3-1）等价于\n",
    "\n",
    "$v_\\pi\\left(\\sfs\\right)=\\E_{\\pi'}\\left[v_\\pi\\left(\\sfS_t\\right)\\mid\\sfS_t=\\sfs\\right]\\le\\E_{\\pi'}\\left[q_\\pi\\left(\\sfS_t,\\sfA_t\\right)\\mid\\sfS_t=\\sfs\\right],\\quad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "其中的期望是针对用策略$\\pi'$生成的轨迹中，选取$\\sfS_t=\\sfs$的那些轨迹而言的。进而有\n",
    "\n",
    "$\\E_{\\pi'}\\left[v_\\pi\\left(\\sfS_{t+\\tau}\\right)|\\sfS_t=\\sfs\\right]=\\E_{\\pi'}\\left[\\E_{\\pi'}\\left[v_\\pi\\left(\\sfS_{t+\\tau}\\right)|\\sfS_{t+\\tau}\\right]|\\sfS_t=\\sfs\\right]\\le\\E_{\\pi'}\\left[\\E_{\\pi'}\\left[q_\\pi\\left(\\sfS_{t+\\tau},\\sfA_{t+\\tau}\\right)|\\sfS_{t+\\tau}\\right]|\\sfS_t=\\sfs\\right]=\\E_{\\pi'}\\left[q_\\pi\\left(\\sfS_{t+\\tau},\\sfA_{t+\\tau}\\right)|\\sfS_t=\\sfs\\right],\\quad\\sfs\\in\\mathcal{S},\\tau=0,1,2,\\ldots$\n",
    "\n",
    "考虑到\n",
    "\n",
    "$\\E_{\\pi'}\\left[q_\\pi\\left(\\sfS_{t+\\tau},\\sfA_{t+\\tau}\\right)|\\sfS_t=\\sfs\\right]=\\E_{\\pi'}\\left[R_{t+\\tau+1}+\\gamma{v_\\pi}\\left(\\sfS_{t+\\tau+1}\\right)|\\sfS_t=\\sfs\\right],\\quad\\sfs\\in\\mathcal{S},\\tau=0,1,2,\\ldots$\n",
    "\n",
    "所以\n",
    "\n",
    "$\\E_{\\pi'}\\left[v_\\pi\\left(\\sfS_{t+\\tau}\\right)|\\sfS_t=\\sfs\\right]\\le\\E_{\\pi'}\\left[R_{t+\\tau+1}+\\gamma{v_\\pi}\\left(\\sfS_{t+\\tau+1}\\right)|\\sfS_t=\\sfs\\right],\\quad\\sfs\\in\\mathcal{S},\\tau=0,1,2,\\ldots$\n",
    "\n",
    "进而有\n",
    "\n",
    "$v_\\pi\\left(\\sfs\\right)=\\E_{\\pi'}\\left[v_\\pi\\left(\\sfS_t\\right)|\\sfS_t=\\sfs\\right]$\n",
    "\n",
    "$\\le\\E_{\\pi'}\\left[R_{t+1}+\\gamma{v_\\pi}\\left(\\sfS_{t+1}\\right)|\\sfS_t=\\sfs\\right]$\n",
    "\n",
    "$\\le\\E_{\\pi'}\\left[R_{t+1}+\\gamma\\E_{\\pi'}\\left[R_{t+2}+\\gamma{v_\\pi}\\left(\\sfS_{t+2}\\right)|\\sfS_t=\\sfs\\right]|\\sfS_t=\\sfs\\right]$\n",
    "\n",
    "$=\\E_{\\pi'}\\left[R_{t+1}+\\gamma{R}_{t+2}+\\gamma^2v_\\pi\\left(\\sfS_{t+2}\\right)\\mid\\sfS_t=\\sfs\\right]$\n",
    "\n",
    "$\\le\\E_{\\pi'}\\left[R_{t+1}+\\gamma{R}_{t+2}+\\gamma^2R_{t+3}+\\gamma^3v_\\pi\\left(\\sfS_{t+3}\\right)|\\sfS_t=\\sfs\\right]$\n",
    "\n",
    "$\\cdots$\n",
    "\n",
    "$\\le\\E_{\\pi'}\\left[R_{t+1}+\\gamma{R}_{t+2}+\\gamma^2{R_{t+3}}+\\gamma^3R_{t+4}+\\cdots|{\\sfS_t}=\\sfs\\right]$\n",
    "\n",
    "$=\\E_{\\pi'}\\left[G_t|\\sfS_t=\\sfs\\right]$\n",
    "\n",
    "$=v_{\\pi'}\\left(\\sfs\\right)$\n",
    "\n",
    "\n",
    "## 第49页第12行\n",
    "\n",
    "这个函数使用`theta`作\n",
    "\n",
    "### 改为\n",
    "\n",
    "这个函数使用`tolerant`作\n",
    "\n",
    "## 第52页代码清单3-10\n",
    "\n",
    "### 代码改为\n",
    "\n",
    "```\n",
    "policy_vi, v_vi = iterate_value(env)\n",
    "print('状态价值函数 =')\n",
    "print(v_vi.reshape(4, 4))\n",
    "print('最优策略 =')\n",
    "print(np.argmax(policy_vi, axis=1).reshape(4, 4))\n",
    "episode_rewards = [play_policy(env, policy_vi) for _ in range(100)]\n",
    "print(\"价值迭代 平均奖励：{}\".format(np.mean(episode_rewards)))\n",
    "```\n",
    "\n",
    "## 第54页第15行\n",
    "\n",
    "$\\frac{1}{c}\\sum\\nolimits_{i=1}^c{g_c}$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\frac{1}{c}\\sum\\nolimits_{i=1}^c{g_i}$\n",
    "\n",
    "\n",
    "## 第56页第10行\n",
    "\n",
    "同时满足$\\left\\{\\alpha_k:k=1,2,\\ldots\\right\\}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\left\\{\\alpha_k:k=1,2,\\ldots\\right\\}$同时满足\n",
    "\n",
    "## 第57页算法4-2第1步，第58页算法4-4第1步（共2处）\n",
    "\n",
    "初始化动作价值估计\n",
    "\n",
    "### 改为\n",
    "\n",
    "初始化状态价值估计\n",
    "\n",
    "## 第60页第20~21行\n",
    "\n",
    "证明：对于某个$\\varepsilon$柔性策略，用\n",
    "\n",
    "$\\pi\\left(\\sfa\\mid\\sfs\\right)=\\left\\{\\begin{matrix}1-\\varepsilon+\\frac{\\varepsilon}{\\left|\\mathcal{A}\\right|},&\\sfa=\\arg\\max_{\\sfa'}q_\\pi\\left(\\sfs,\\sfa'\\right)\\\\ \\frac{\\varepsilon}{\\left|\\mathcal{A}\\right|},&\\sfa\\ne\\arg\\max_\\sfa'q_\\pi\\left(\\sfs,\\sfa'\\right)\\end{matrix}\\right.$\n",
    "\n",
    "改进后的策略为$\\pi'$，则有\n",
    "\n",
    "${q_\\pi}\\left(\\sfs,\\sfa\\right)=\\sum\\limits_\\sfa{\\pi'\\left(\\sfa|\\sfs\\right){q_\\pi}\\left(\\sfs,\\sfa\\right)}=\\frac{\\varepsilon}{{\\left|{{\\mathcal{A}}\\left(\\sfs\\right)}\\right|}}\\sum\\limits_\\sfa{{q_\\pi }\\left(\\sfs,\\sfa\\right)}+\\left(1-\\varepsilon\\right)\\mathop{\\max }\\limits_\\sfa{q_\\pi}\\left(\\sfs,\\sfa\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "证明：考虑在$\\varepsilon$柔性策略$\\pi$上进行如下方式改进得到的$\\pi$柔性策略$\\pi'$：\n",
    "\n",
    "$\\pi'\\left(\\sfa|\\sfs\\right)=\\left\\{\\begin{matrix}1-\\varepsilon+\\frac{\\varepsilon}{\\left|\\mathcal{A}\\right|},&\\sfa=\\arg\\max_{\\sfa'}q_\\pi\\left(\\sfs,\\sfa'\\right)\\\\ \\frac{\\varepsilon}{\\left|\\mathcal{A}\\right|},&\\sfa\\ne\\arg\\max_\\sfa'q_\\pi\\left(\\sfs,\\sfa'\\right)\\end{matrix}\\right.$\n",
    "\n",
    "在3.2.2节策略改进定理的证明过程中，我们已经证明，只要\n",
    "\n",
    "$\\sum\\limits_\\sfa{\\pi'\\left(\\sfa|\\sfs\\right){q_\\pi}\\left(\\sfs,\\sfa\\right)}\\ge{v_\\pi}\\left(\\sfs\\right),\\quad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "就有$\\pi\\le\\pi'$。接下来验证上述不等式。考虑到\n",
    "\n",
    "$\\sum\\limits_\\sfa\\pi'\\left(\\sfa\\mid\\sfs\\right)q_\\pi\\left(\\sfs,\\sfa\\right)=\\frac{\\varepsilon}{\\left|\\mathcal{A}\\left(\\sfs\\right)\\right|}\\sum\\limits_\\sfa{q_\\pi}\\left(\\sfs,\\sfa\\right)+\\left(1-\\varepsilon\\right)\\max\\limits_\\sfa{q_\\pi}\\left(\\sfs,\\sfa\\right)$\n",
    "\n",
    "\n",
    "## 第61页第4行\n",
    "\n",
    "$q_{\\pi'}\\left(\\sfs,\\sfa\\right)=\\frac{\\varepsilon}{\\left|\\mathcal{A}\\left(\\sfs\\right)\\right|}\\sum\\limits_\\sfa{q_\\pi\\left(\\sfs,\\sfa\\right)}+\\left(1-\\varepsilon\\right)\\mathop{\\max}\\limits_{\\sfa}q_\\pi\\left(\\sfs,\\sfa\\right)$\n",
    "\n",
    "## 改为\n",
    "\n",
    "$\\sum\\limits_{\\sfa}{\\pi'\\left(\\sfa|\\sfs\\right)q_\\pi\\left(\\sfs,\\sfa\\right)}=\\frac{\\varepsilon}{\\left|\\mathcal{A}\\left(\\sfs\\right)\\right|}\\sum\\limits_\\sfa{q_\\pi\\left(\\sfs,\\sfa\\right)}+\\left(1-\\varepsilon\\right)\\mathop{\\max}\\limits_{\\sfa}q_\\pi\\left(\\sfs,\\sfa\\right)$\n",
    "\n",
    "\n",
    "## 第63页第13行\n",
    "\n",
    "回报是等概率出现的是。\n",
    "\n",
    "### 改为\n",
    "\n",
    "回报是等概率出现的。\n",
    "\n",
    "## 第64页第3行\n",
    "\n",
    "$v\\leftarrow v+\\frac{1}{c}\\left(g-v\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$v\\leftarrow v+\\frac{\\rho}{c}\\left(g-v\\right)$\n",
    "\n",
    "## 第64页第9行\n",
    "\n",
    "算法4-7给出了每次方法加权重要性采样\n",
    "\n",
    "### 改为\n",
    "\n",
    "算法4-7给出了每次访问加权重要性采样\n",
    "\n",
    "## 第64页算法4-7第2.4.2步\n",
    "\n",
    "$w\\left[G-q\\left(\\sfS_t,\\sfA_t\\right)\\right]^2$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\rho\\left[G-q\\left(\\sfS_t,\\sfA_t\\right)\\right]^2$\n",
    "\n",
    "\n",
    "## 第65页算法4-8第2.2步\n",
    "\n",
    "$\\sfS_0,\\sfA_0,R_1,\\sfS_1,\\sfA_1,R_1,\\ldots,\\sfS_{T-1},\\sfA_{T-1},R_T,\\sfS_T$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\sfS_0,\\sfA_0,R_1,\\sfS_1,\\sfA_1,R_2,\\ldots,\\sfS_{T-1},\\sfA_{T-1},R_T,\\sfS_T$\n",
    "\n",
    "## 第67页第6行\n",
    "\n",
    "是有将1张A牌计算为11点。\n",
    "\n",
    "### 改为\n",
    "\n",
    "是否将1张A牌计算为11点。\n",
    "\n",
    "## 第71页第7行\n",
    "\n",
    "最优价值和最优价值函数\n",
    "\n",
    "### 改为\n",
    "\n",
    "最优策略和最优价值函数\n",
    "\n",
    "## 第72页4.3.4节正文第1行，第73页正文第0行\n",
    "\n",
    "`evaluate_action_monte_carlo_importance_resample`\n",
    "\n",
    "### 改为\n",
    "\n",
    "`evaluate_monte_carlo_importance_sample`\n",
    "\n",
    "## 第72页代码清单4-7第0行，第73页正文第7行\n",
    "\n",
    "`evaluate_monte_carlo_importance_resample`\n",
    "\n",
    "### 改为\n",
    "\n",
    "`evaluate_monte_carlo_importance_sample`\n",
    "\n",
    "## 第73页代码清单4-8第0行，第74页正文第0行，第74页正文第3行\n",
    "\n",
    "`monte_carlo_importance_resample`\n",
    "\n",
    "改为\n",
    "\n",
    "`monte_carlo_importance_sample`\n",
    "\n",
    "## 第75页第8行\n",
    "\n",
    "$\\rho_{t:T-1}=\\prod\\limits_{k=t}^{T-1}{\\frac{{\\pi\\left(\\sfA_t|\\sfS_t\\right)}}{{b\\left(\\sfA_t|\\sfS_t\\right)}}}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\rho_{t:T-1}=\\prod\\limits_{\\tau=t}^{T-1}{\\frac{\\pi\\left(\\sfA_\\tau|\\sfS_\\tau\\right)}{{b\\left(\\sfA_\\tau|\\sfS_\\tau\\right)}}}$\n",
    "\n",
    "\n",
    "## 第77页第2行\n",
    "\n",
    "$U_{t:t+1}^{\\left(q\\right)}=R_{t+1}+\\gamma q\\left(\\sfS_{t+1},\\sfA_{t+n}\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U_{t:t+1}^{\\left(q\\right)}=R_{t+1}+\\gamma q\\left(\\sfS_{t+1},\\sfA_{t+1}\\right)$\n",
    "\n",
    "\n",
    "## 第77页第15行\n",
    "\n",
    "$U_{t:t+1}^{\\left(q\\right)}=\\left\\{\\begin{matrix}{R_{t+1}+\\gamma{R}_{t+2}+\\cdots+\\gamma^{n-1}{R}_{t+n}+{\\gamma^n}q\\left(\\sfS_{t+1},\\sfA_{t+n}\\right),}&t+n<T,\\\\R_{t+1}+\\gamma{R}_{t+2}+\\cdots+\\gamma^{T-t-1}R_T,&t+n\\ge{T}.\\end{matrix}\\right.$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U_{t:t+n}^{\\left(q\\right)}=\\left\\{\\begin{matrix}{R_{t+1}+\\gamma{R}_{t+2}+\\cdots+\\gamma^{n-1}{R}_{t+n}+{\\gamma^n}q\\left(\\sfS_{t+n},\\sfA_{t+n}\\right),}&t+n<T,\\\\R_{t+1}+\\gamma{R}_{t+2}+\\cdots+\\gamma^{T-t-1}R_T,&t+n\\ge{T}.\\end{matrix}\\right.$\n",
    "\n",
    "## 第79页算法5-2第2.1步\n",
    "\n",
    "2.1（初始化状态动作对）\n",
    "\n",
    "### 改为\n",
    "2.1（初始化状态）\n",
    "\n",
    "## 第81页算法5-4输出\n",
    "\n",
    "输出：动作价值估计\n",
    "\n",
    "### 改为\n",
    "\n",
    "输出：状态价值估计\n",
    "\n",
    "## 第81页算法5-4第2.2.1步\n",
    "\n",
    "$U\\leftarrow{R}_{t+1}+\\gamma{R}_{t+2}+\\cdots+\\gamma^{n-1}R_{t+n}+{\\gamma^n}v\\left(\\sfS_{t+n},{\\gamma^n}v\\sfS_{t+n}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U\\leftarrow{R}_{t+1}+\\gamma{R}_{t+2}+\\cdots+\\gamma^{n-1}R_{t+n}+{\\gamma^n}v\\left(\\sfS_{t+n}\\right)$\n",
    "\n",
    "\n",
    "## 第84页算法5-8步骤2.2\n",
    "\n",
    "前一个步骤2.2的编号应为2.1。\n",
    "\n",
    "## 第85页第11行\n",
    "\n",
    "$=\\prod\\limits_{\\tau=t}^{t+n-1}\\frac{\\pi\\left(\\sfA_\\tau|\\sfS_\\tau\\right)}{b\\left(\\sfA_\\tau|\\sfS_\\tau\\right)}$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$=\\prod\\limits_{\\tau=t+1}^{t+n-1}\\frac{\\pi\\left(\\sfA_\\tau|\\sfS_\\tau\\right)}{b\\left(\\sfA_\\tau|\\sfS_\\tau\\right)}$\n",
    "\n",
    "\n",
    "## 第85页算法5-10输出\n",
    "\n",
    "若是最优策略控制则还是输出策略\n",
    "\n",
    "### 改为\n",
    "\n",
    "若是最优策略控制则还要输出策略\n",
    "\n",
    "## 第87页算法5-11第2.1步\n",
    "\n",
    "初始化状态动作对\n",
    "\n",
    "### 改为\n",
    "\n",
    "初始化状态\n",
    "\n",
    "## 第87页第14行\n",
    "\n",
    "使得从$\\sfs$<sub>状态</sub>更倾向于选择\n",
    "\n",
    "### 改为\n",
    "\n",
    "使得从$\\sfs$<sub>开始</sub>更倾向于选择\n",
    "\n",
    "## 第88页算法5-12第2.1步\n",
    "\n",
    "2.1（初始化状态动作对）\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.1（初始化状态）\n",
    "\n",
    "## 第88页算法5-12第2.2.4步\n",
    "\n",
    "$U\\leftarrow{R}+\\gamma{q^{\\left(1-i\\right)}}\\left(\\sfS_{t+1},\\arg{{\\max}_\\sfa}{q^{\\left(i\\right)}}\\left(\\sfS_{t+1},\\sfa\\right)\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U\\leftarrow{R}+\\gamma{q^{\\left(1-i\\right)}}\\left(\\sfS',\\arg{\\max_\\sfa}{q^{\\left(i\\right)}}\\left(\\sfS',\\sfa\\right)\\right)$\n",
    "\n",
    "\n",
    "## 第89页图5-3子图a和子图b（共2处）\n",
    "\n",
    "$\\left(1-\\lambda\\right)\\lambda^{T-t-1}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\lambda^{T-t-1}$\n",
    "\n",
    "\n",
    "## 第90页第8行\n",
    "\n",
    "时序差分目标$U_{t-n:t}$的权重是$\\left(1-\\lambda\\right)\\lambda^n$。\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "时序差分目标$U_{t-n:t}$的权重是$\\left(1-\\lambda\\right)\\lambda^{n-1}$。\n",
    "\n",
    "\n",
    "## 第90页第13行\n",
    "\n",
    "对的单步自益结果\n",
    "\n",
    "### 改为\n",
    "\n",
    "的单步自益结果\n",
    "\n",
    "## 第90页第20行\n",
    "\n",
    "$U_{\\tau:t}=R_{\\tau+1}+\\cdots+\\gamma^{t-\\tau-1}U_{t:t+1}$，所以$U_{t:t+1}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U_{\\tau:t}=R_{\\tau+1}+\\cdots+\\gamma^{t-\\tau-1}U_{t-1:t}$，所以$U_{t-1:t}$\n",
    "\n",
    "\n",
    "\n",
    "## 第91页算法5-13\n",
    "\n",
    "### 将第1步的\n",
    "\n",
    "初始化资格迹$e\\left(\\sfs,\\sfa\\right)\\leftarrow{0},\\sfs\\in\\mathcal{S},\\sfa\\in\\mathcal{A}\\left(\\sfs\\right)$。\n",
    "\n",
    "### 移动到2.1步\n",
    "\n",
    "\n",
    "## 第91页算法5-13第2.2.2步~2.2.5步\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.2.2 根据输入策略$\\pi\\left(\\cdot|\\sfS'\\right)$或是迭代的最优价值函数$q\\left(\\sfS',\\cdot\\right)$确定动作$\\sfA'$；\n",
    "\n",
    "2.2.3（更新资格迹）$e\\left(\\sfs,\\sfa\\right)\\leftarrow\\gamma\\lambda{e}\\left(\\sfs,\\sfa\\right),\\sfs\\in\\mathcal{S},\\sfa\\in\\mathcal{A}\\left(\\sfs\\right)$，$e\\left(\\sfS,\\sfA\\right)\\leftarrow{1}+\\beta{e}\\left(\\sfS,\\sfA\\right)$；\n",
    "\n",
    "2.2.4（计算回报的估计值）$U\\leftarrow{R}+\\gamma{q}\\left(\\sfS',\\sfA'\\right)$；\n",
    "\n",
    "2.2.5（更新价值）$q\\left(\\sfs,\\sfa\\right)\\leftarrow{q}\\left(\\sfs,\\sfa\\right)+\\alpha{e}\\left(\\sfs,\\sfa\\right)\\left[U-q\\left(\\sfS,\\sfA\\right)\\right],\\sfs\\in\\mathcal{S},\\sfa\\in\\mathcal{A}\\left(\\sfs\\right)$;\n",
    "\n",
    "\n",
    "## 第92页算法5-14第1步\n",
    "\n",
    "### 删去\n",
    "\n",
    "初始化资格迹$c\\left(\\sfs\\right)\\leftarrow{0},\\sfs\\in\\mathcal{S}$。\n",
    "\n",
    "\n",
    "## 第92页算法5-14第2.1步\n",
    "\n",
    "2.1（初始化状态动作对）选择状态$v\\left(\\sfs\\right)$。\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.1（初始化状态和资格迹）选择状态$\\sfS$，初始化资格迹$e\\left(\\sfs\\right)\\leftarrow{0},\\sfs\\in\\mathcal{S}$。\n",
    "\n",
    "\n",
    "## 第92页算法5-14第2.2.5步\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.2.5（更新价值）$v\\left(\\sfs\\right)\\leftarrow{v}\\left(\\sfs\\right)+\\alpha{e}\\left(\\sfs\\right)\\left[U-v\\left(\\sfS\\right)\\right],\\sfs\\in\\mathcal{S}$。\n",
    "\n",
    "\n",
    "## 第92页第11行、第93页第12行、第94页代码清单5-1第1行\n",
    "\n",
    "Taxi-v2\n",
    "\n",
    "### 改为\n",
    "\n",
    "Taxi-v3\n",
    "\n",
    "（共3处。Gym 0.15.2版本Taxi-v2变更地图为Taxi-v3）\n",
    "\n",
    "## 第93行图5-5\n",
    "\n",
    "```\n",
    "+---------+\n",
    "|R: | : :G|\n",
    "| : : : : |\n",
    "| : : : : |\n",
    "| | : | : |\n",
    "|Y| : |B: |\n",
    "+---------+\n",
    "```\n",
    "\n",
    "### 改为\n",
    "\n",
    "```\n",
    "+---------+\n",
    "|R: | : :G|\n",
    "| : | : : |\n",
    "| : : : : |\n",
    "| | : | : |\n",
    "|Y| : |B: |\n",
    "+---------+\n",
    "```\n",
    "\n",
    "（Gym 0.15.2版本Taxi-v2变更地图为Taxi-v3）\n",
    "\n",
    "## 第96页代码清单5-6第14行\n",
    "\n",
    "```\n",
    "        v = (self.q[next_state].sum() * self.epsilon + \\\n",
    "```\n",
    "\n",
    "### 改为\n",
    "\n",
    "```\n",
    "        v = (self.q[next_state].mean() * self.epsilon + \\\n",
    "```\n",
    "\n",
    "## 第98页第2行\n",
    "\n",
    "`DoubleQLearnignAgent`类\n",
    "\n",
    "### 改为\n",
    "\n",
    "`DoubleQLearningAgent`类\n",
    "\n",
    "## 第99页代码清单5-12\n",
    "\n",
    "### 改为\n",
    "\n",
    "```\n",
    "class SARSALambdaAgent(SARSAAgent):\n",
    "    def __init__(self, env, lambd=0.5, beta=1.,\n",
    "            gamma=0.9, learning_rate=0.1, epsilon=.01):\n",
    "        super().__init__(env, gamma=gamma, learning_rate=learning_rate,\n",
    "                epsilon=epsilon)\n",
    "        self.lambd = lambd\n",
    "        self.beta = beta\n",
    "        self.e = np.zeros((env.observation_space.n, env.action_space.n))\n",
    "        \n",
    "    def learn(self, state, action, reward, next_state, done, next_action):\n",
    "        # 更新资格迹\n",
    "        self.e *= (self.lambd * self.gamma)\n",
    "        self.e[state, action] = 1. + self.beta * self.e[state, action]\n",
    "        \n",
    "        # 更新价值\n",
    "        u = reward + self.gamma * \\\n",
    "                self.q[next_state, next_action] * (1. - done)\n",
    "        td_error = u - self.q[state, action]\n",
    "        self.q += self.learning_rate * self.e * td_error\n",
    "        \n",
    "        # 为下一回合初始化资格迹\n",
    "        if done:\n",
    "            self.e *= 0.\n",
    "\n",
    "agent = SARSALambdaAgent(env)\n",
    "```\n",
    "\n",
    "## 第100页第12行\n",
    "\n",
    "$U_t^{\\left(i\\right)}=R_{t+1}+\\gamma{q}^{\\left(i\\right)}\\left(\\sfS_{t+1},\\arg\\max_{\\sfa}q^{\\left(1-i\\right)}\\left(\\sfS_{t+1},\\sfa\\right)\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U_t^{\\left(i\\right)}=R_{t+1}+\\gamma{q}^{\\left(1-i\\right)}\\left(\\sfS_{t+1},\\arg\\max_{\\sfa}q^{\\left(i\\right)}\\left(\\sfS_{t+1},\\sfa\\right)\\right)$\n",
    "\n",
    "\n",
    "## 第100页第16行\n",
    "\n",
    "形式为$U_t=R_{t+1}+q\\left(\\sfS_{t+1},\\sfA_{t+1}\\right)$。\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "形式为$U_t=R_{t+1}+\\gamma{q}\\left(\\sfS_{t+1},\\sfA_{t+1}\\right)$。\n",
    "\n",
    "\n",
    "\n",
    "## 第102页第5行\n",
    "\n",
    "回合的损失为$\\sum\\nolimits_{t=0}^T{{\\left[G_t-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]}^2}$。如果我们沿着$\\sum\\nolimits_{t=0}^T{{\\left[G_t-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]}^2}$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "回合的损失为$\\sum\\nolimits_{t=0}^{T-1}{{\\left[G_t-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]}^2}$。如果我们沿着$\\sum\\nolimits_{t=0}^{T-1}{{\\left[G_t-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]}^2}$\n",
    "\n",
    "\n",
    "## 第103页第5行\n",
    "\n",
    "$\\sum\\nolimits_{t=0}^T{{\\left[U_t-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]}^2}$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\sum\\nolimits_{t=0}^{T-1}{{\\left[U_t-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]}^2}$\n",
    "\n",
    "\n",
    "## 第104页算法6-3第2.2.4步\n",
    "\n",
    "$\\mathbf{w}\\leftarrow\\mathbf{w}+\\alpha\\left[G-q\\left(\\sfS,\\sfA;\\mathbf{w}\\right)\\right]\\nabla{q}\\left(\\sfS,\\sfA;\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\mathbf{w}\\leftarrow\\mathbf{w}+\\alpha\\left[U-q\\left(\\sfS,\\sfA;\\mathbf{w}\\right)\\right]\\nabla{q}\\left(\\sfS,\\sfA;\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "## 第104页算法6-4第2.1步\n",
    "\n",
    "2.1（初始化状态动作对）选择状态$\\sfS$，再根据输入策略$\\pi$选择动作$\\sfA$。\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.1（初始化状态）选择状态$\\sfS$。\n",
    "\n",
    "## 第104页算法6-4第2.2.3步、第106页算法6-7第2.2.3步（共2处）\n",
    "\n",
    "删去此步\n",
    "\n",
    "## 第104页算法6-4第2.2.4步、第106页算法6-7第2.2.4步（共2处）\n",
    "\n",
    "如果是动作价值评估\n",
    "\n",
    "### 改为\n",
    "\n",
    "如果是状态价值评估\n",
    "\n",
    "## 第104页第5行\n",
    "\n",
    "算法6-5\n",
    "\n",
    "### 改为\n",
    "\n",
    "算法6-3\n",
    "\n",
    "## 第105页算法6-5\n",
    "\n",
    "删去该算法\n",
    "\n",
    "## 第105页第9~10行\n",
    "\n",
    "$\\mathbf{w}\\leftarrow{\\mathbf{w}}+\\alpha\\left[U-q\\left(\\sfS_t,\\sfA_t\\right)\\right]\\mathbf{z}$, 更新动作价值\n",
    "\n",
    "$\\mathbf{w}\\leftarrow{\\mathbf{w}}+\\alpha\\left[U-v\\left(\\sfS_t\\right)\\right]\\mathbf{z}$, 更新状态价值\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\mathbf{w}\\leftarrow\\mathbf{w}+\\alpha\\left[U-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]\\mathbf{z}$, 更新动作价值\n",
    "\n",
    "$\\mathbf{w}\\leftarrow\\mathbf{w}+\\alpha\\left[U-v\\left(\\sfS_t;\\mathbf{w}\\right)\\right]\\mathbf{z}$, 更新状态价值\n",
    "\n",
    "\n",
    "\n",
    "## 第106页算法6-7第2.1步\n",
    "\n",
    "2.1（初始化状态动作对）选择状态$\\sfS$，再根据输入策略$\\pi$选择动作$\\sfA$。\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.1（初始化状态和资格迹）选择状态$\\sfS$，初始化资格迹$\\mathbf{z}\\leftarrow\\mathbf{0}$。\n",
    "\n",
    "\n",
    "## 第109页算法6-8输出和第1步\n",
    "\n",
    "$q\\left(\\sfs_t,\\sfa_t;\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$q\\left(\\sfs,\\sfa;\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "## 第109页算法6-8第2.2步\n",
    "\n",
    "决定确定性策略\n",
    "\n",
    "### 改为\n",
    "\n",
    "决定策略\n",
    "\n",
    "## 第111页算法6-9第2.1步、第113页算法6-10第2.1步（共2处）\n",
    "\n",
    "2.1（初始化状态动作对）\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.1（初始化状态）\n",
    "\n",
    "## 第112页第12行\n",
    "\n",
    "$P_i=\\frac{p_i}{\\sum_{k}{p_k}}$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\frac{p_i}{\\sum_{k}{p_k}}$\n",
    "\n",
    "\n",
    "## 第112页第14行\n",
    "\n",
    "其中$\\delta_i$是时序差分误差，\n",
    "\n",
    "### 改为\n",
    "\n",
    "其中$\\delta_i$是时序差分误差（定义为$\\delta_t=U_t-q\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)$或$\\delta_t=U_t-v\\left(\\sfS_t;\\mathbf{w}\\right)$），\n",
    "\n",
    "\n",
    "## 第118页\n",
    "\n",
    "作者注：\n",
    "\n",
    "砖瓦编码是一种历史悠久的特征构造方法，可用于回归、分类等问题。目前学术界倾向于用神经网络代替砖瓦编码来构造特征。由于砖瓦编码和强化学习没有直接关联，本书没有用过多的篇幅介绍砖瓦编码。\n",
    "\n",
    "实际使用砖瓦编码时，不需要精确计算砖瓦的数量，常随意的大致估计砖瓦的数量作为特征数。如果设置的特征数大于真实的砖瓦数量，那么有些特征永远不会取到，有些浪费；如果设置的特征数小于真实的砖瓦数量，那么有多个砖瓦需要共享特征，具体逻辑可以见代码清单6-3中“冲突处理”部分。这些浪费和冲突往往不会造成明显的性能损失。\n",
    "\n",
    "第118页砖瓦数计算：每个大网格的网格宽度刚好是整个取值范围的1/8。所以，第0层大网格每个维度有8个大网格；第1~7层由于有偏移，每个维度需要有9个大网格才能覆盖整个取值范围。第117页图6-3b的情况略有不同：这个图中每个维度的取值范围不是大网格的长度的整数倍。所以有些层偏移后，不需要更多的大网格也可以覆盖整个取值范围。\n",
    "\n",
    "\n",
    "## 第126页第10行\n",
    "\n",
    "再配合其他一些容易活动的\n",
    "\n",
    "### 改为\n",
    "\n",
    "再配合其他一些容易获得的\n",
    "\n",
    "\n",
    "## 第126页最后一行\n",
    "\n",
    "$=\\sum\\limits_\\sfs\\Pr\\left[\\sfS_t=\\sfs\\right]\\nabla{v}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_t\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$=\\sum\\limits_\\sfs\\Pr\\left[\\sfS_t=\\sfs\\right]\\nabla{v}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfs\\right)$\n",
    "\n",
    "\n",
    "## 第127页第18行\n",
    "\n",
    "$=\\E\\left[\\gamma^t{q}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_t,\\sfA_t\\right)\\nabla\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)\\right]$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$=\\E\\left[\\gamma^t{q}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_t,\\sfA_t\\right)\\nabla\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)\\right]$\n",
    "\n",
    "\n",
    "## 第127页第20行\n",
    "\n",
    "$\\E\\left[\\sum\\limits_\\sfa{\\gamma^t}q_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_t,\\sfa\\right)\\nabla\\pi\\left(\\sfa|\\sfS_t;\\bftheta\\right)\\right]=\\E\\left[\\gamma^t{q}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_t,\\sfA_t\\right)\\nabla\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)\\right]=\\E\\left[\\gamma^t{G_t}\\nabla\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)\\right]$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\E\\left[\\sum\\limits_\\sfa{\\gamma^t}q_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_t,\\sfa\\right)\\nabla\\pi\\left(\\sfa|\\sfS_t;\\bftheta\\right)\\right]=\\E\\left[\\gamma^t{q}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_t,\\sfA_t\\right)\\nabla\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)\\right]=\\E\\left[\\gamma^t{G_t}\\nabla\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)\\right]$\n",
    "\n",
    "\n",
    "\n",
    "## 第128页算法7-1输入和第130页算法7-2输入（共2处）\n",
    "\n",
    "输入：环境（无数学描述）、策略$\\pi$。\n",
    "\n",
    "### 改为\n",
    "\n",
    "输入：环境（无数学描述）。\n",
    "\n",
    "## 第128页算法7-1第2步和第130页算法7-2第2步（共2处）\n",
    "\n",
    "2.（时序差分更新）\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.（回合更新）\n",
    "\n",
    "\n",
    "## 第129页第10~11行\n",
    "\n",
    "$=\\sum\\limits_\\sfa{\\gamma^t\\left(G_t-B\\left(\\sfS_t\\right)\\right)\\nabla\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)}$\n",
    "\n",
    "$=\\sum\\limits_\\sfa{\\gamma^tG_t\\nabla\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)}$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$=\\sum\\limits_\\sfa{\\gamma^t\\left(G_t-B\\left(\\sfS_t\\right)\\right)\\nabla\\pi\\left(\\sfa|\\sfS_t;\\bftheta\\right)}$\n",
    "\n",
    "$=\\sum\\limits_\\sfa{\\gamma^tG_t\\nabla\\pi\\left(\\sfa|\\sfS_t;\\bftheta\\right)}$\n",
    "\n",
    "\n",
    "## 第129页第24行\n",
    "\n",
    "值函数的估计是可以得到的。\n",
    "\n",
    "### 改为\n",
    "\n",
    "值函数的估计。\n",
    "\n",
    "## 第130页算法7-2第2.3.3步\n",
    "\n",
    "更新$\\bftheta$以减小$-\\gamma^t\\left[G-v\\left(\\sfS_t;\\mathbf{w}\\right)\\right]\\nabla\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)$\n",
    "\n",
    "\n",
    "## 改为\n",
    "\n",
    "更新$\\bftheta$以减小$-\\gamma^t\\left[G-v\\left(\\sfS_t;\\mathbf{w}\\right)\\right]\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)$\n",
    "\n",
    "\n",
    "## 第130页第4行\n",
    "\n",
    "$\\E\\left[-2\\gamma^t\\left(G_t-B\\left(\\sfS_t\\right)\\right){\\left[ {\\nabla\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)} \\right]}^2\\right]$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\E\\left[-2\\gamma^{2t}\\left(G_t-B\\left(\\sfS_t\\right)\\right){\\left[ {\\nabla\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)} \\right]}^2\\right]$\n",
    "\n",
    "\n",
    "## 第130页第11行\n",
    "\n",
    "不过，由于梯度$\\nabla\\ln\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)$并不会预先知道，所以实际应用时\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "但是，在实际应用中，无法事先知道这个值，所以\n",
    "\n",
    "## 第139页第17行\n",
    "\n",
    "减了基线$b\\left(\\sfs\\right)=v_\\pi\\left(\\sfs\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "减去基线$B\\left(\\sfs\\right)=v_\\pi\\left(\\sfs\\right)$\n",
    "\n",
    "\n",
    "## 第139页第18行\n",
    "\n",
    "加上基线$b\\left(\\sfs\\right)=v_\\pi\\left(\\sfs\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "减去基线$B\\left(\\sfs\\right)=v_\\pi\\left(\\sfs\\right)$\n",
    "\n",
    "## 第139页第13行和第18行（2处）\n",
    "\n",
    "$R_t+\\gamma{v_\\pi}\\left(\\sfS_{t+1}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$R_{t+1}+\\gamma{v_\\pi}\\left(\\sfS_{t+1}\\right)$\n",
    "\n",
    "\n",
    "## 第140页第4行\n",
    "\n",
    "$R_{t}+v_\\pi\\left(\\sfS_{t+1};\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$R_{t+1}+\\gamma{v_\\pi}\\left(\\sfS_{t+1};\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "## 第140页算法8-1第2.2步\n",
    "\n",
    "2.2（决定初始动作）用$\\pi\\left(\\cdot|\\sfS;\\bftheta\\right)$得到动作$\\sfA$；\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.2（决定初始状态动作对）选择状态$\\sfS$，用$\\pi\\left(\\cdot|\\sfS;\\bftheta\\right)$得到动作$\\sfA$；\n",
    "\n",
    "\n",
    "## 第140页算法8-1第2.3.1步\n",
    "\n",
    "得到采样$R$和\n",
    "\n",
    "### 改为\n",
    "\n",
    "得到奖励$R$和\n",
    "\n",
    "## 第141页第4行和第5行\n",
    "\n",
    "$R_t+\\gamma{v_\\pi}\\left(\\sfS_{t+1};\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$R_{t+1}+\\gamma{v_\\pi}\\left(\\sfS_{t+1};\\mathbf{w}\\right)$\n",
    "\n",
    "## 第141页算法8-2的第2.2步至第2.3.3步，第143页算法8-4的第2.2步至第2.3.3步\n",
    "\n",
    "2.2（决定初始动作）用$\\pi\\left(\\cdot|\\sfS;\\bftheta\\right)$得到动作$\\sfA$；\n",
    "\n",
    "2.3 如果回合未结束，执行以下操作：\n",
    "\n",
    "2.3.1（采样）根据状态$\\sfS$和动作$\\sfA$得到采样$R$和下一状态$\\sfS'$；\n",
    "\n",
    "2.3.2（执行）用$\\pi\\left(\\cdot|\\sfS';\\bftheta\\right)$得到动作$\\sfA'$\n",
    "\n",
    "2.3.3（估计回报）$U\\leftarrow{R}+\\gamma{q}\\left(\\sfS',\\sfA';\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.2（决定初始状态）选择状态$\\sfS$；\n",
    "\n",
    "2.3 如果回合未结束，执行以下操作：\n",
    "\n",
    "2.3.1（采样）用$\\pi\\left(\\cdot|\\sfS;\\bftheta\\right)$得到动作$\\sfA$；\n",
    "\n",
    "2.3.2（执行）执行动作$\\sfA$，得到奖励$R$和观测$\\sfS'$；\n",
    "\n",
    "2.3.3（估计回报）$U\\leftarrow{R}+\\gamma{v}\\left(\\sfS';\\mathbf{w}\\right)$\n",
    "\n",
    "\n",
    "## 第141页算法8-2的第2.3.7步，第143页算法8-4的第2.3.9步\n",
    "\n",
    "$\\sfS\\leftarrow\\sfS'$，$\\sfA\\leftarrow\\sfA'$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\sfS\\leftarrow\\sfS'$\n",
    "\n",
    "\n",
    "## 第142页算法8.3第2.3.2步\n",
    "\n",
    "作者注：这里的更新式子遵循了论文原文而没有考虑累积折扣。推导出现折扣是正确的；更新时考虑折扣也是正确和合理的。\n",
    "\n",
    "\n",
    "## 第143页算法8-4\n",
    "\n",
    "### 将第1步的\n",
    "\n",
    "初始化资格迹$\\mathbf{z}^{\\left(\\bftheta \\right)}\\leftarrow\\mathbf{0}$，$\\mathbf{z}^{\\left(\\mathbf{w}\\right)}\\leftarrow\\mathbf{0}$。\n",
    "\n",
    "\n",
    "### 移动到第2.1步\n",
    "\n",
    "## 第143页算法8-4第2.2步\n",
    "\n",
    "2.2（决定初始动作）用$\\pi\\left(\\cdot|\\sfS;\\bftheta\\right)$得到动作$\\sfA$；\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.2（决定初始状态）选择状态$\\sfS$；\n",
    "\n",
    "\n",
    "## 第144页第5~9行\n",
    "\n",
    "$\\E_{\\pi\\left(\\bftheta\\right)}\\left[\\sum\\limits_{t=0}^{+\\infty}{\\gamma^t{a_{\\pi_k}}\\left(\\sfS_t,\\sfA_t\\right)}\\right]$\n",
    "\n",
    "$=\\E_{\\pi\\left(\\bftheta\\right)}\\left[\\sum\\limits_{t=0}^{+\\infty}{\\gamma^t\\left(R_t+\\gamma{v_{\\pi\\left(\\bftheta_k\\right)}}\\left(\\sfS_{t+1}\\right)-v_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_t\\right)\\right)}\\right]$\n",
    "\n",
    "$=\\E_{\\pi\\left(\\bftheta\\right)}\\left[-v_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_0\\right)+\\sum\\limits_{t=0}^{+\\infty}{\\gamma^tR_t}\\right]$\n",
    "\n",
    "$=-\\E_{\\sfS_0}\\left[v_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_0\\right)\\right]+\\E_{\\pi\\left(\\bftheta\\right)}\\left[\\sum\\limits_{t=0}^{+\\infty}{\\gamma^tR_t}\\right]$\n",
    "\n",
    "$=-\\E_{\\pi\\left(\\bftheta_k\\right)}\\left[G_0\\right]+\\E_{\\pi\\left(\\bftheta\\right)}\\left[G_0\\right].$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\E_{\\pi\\left(\\bftheta\\right)}\\left[\\sum\\limits_{t=0}^{+\\infty}{\\gamma^{t}a_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_t,\\sfA_t\\right)}\\right]$\n",
    "\n",
    "$=\\E_{\\pi\\left(\\bftheta\\right)}\\left[\\sum\\limits_{t=0}^{+\\infty}{\\gamma^t\\left(R_{t+1}+\\gamma{v}_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_{t+1}\\right)-v_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_t\\right)\\right)}\\right]$\n",
    "\n",
    "$=\\E_{\\pi\\left(\\bftheta\\right)}\\left[-v_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_0\\right)+\\sum\\limits_{t=0}^{+\\infty}{\\gamma^tR_{t+1}}\\right]$\n",
    "\n",
    "$=-\\E_{\\sfS_0}\\left[v_{\\pi\\left(\\bftheta_k\\right)}\\left(\\sfS_0\\right)\\right]+\\E_{\\pi\\left(\\bftheta\\right)}\\left[\\sum\\limits_{t=0}^{+\\infty}{\\gamma^t{R_{t+1}}}\\right]$\n",
    "\n",
    "$=-\\E_{\\pi\\left(\\bftheta_k\\right)}\\left[G_0\\right]+\\E_{\\pi\\left(\\bftheta\\right)}\\left[G_0\\right].$\n",
    "\n",
    "\n",
    "## 第145页算法8-5第2.3步\n",
    "\n",
    "更新$\\bftheta$以减小\n",
    "\n",
    "### 改为\n",
    "\n",
    "更新$\\bftheta$以增大\n",
    "\n",
    "\n",
    "## 第145页算法8-5第2.4步\n",
    "\n",
    "最小化$\\left[G_t-v\\left(\\sfS_t,\\sfA_t;\\mathbf{w}\\right)\\right]^2$\n",
    "\n",
    "### 改为\n",
    "\n",
    "最小化$\\left[G_t-v\\left(\\sfS_t;\\mathbf{w}\\right)\\right]^2$\n",
    "\n",
    "\n",
    "## 第148页第7行\n",
    "\n",
    "minimize\n",
    "\n",
    "### 改为\n",
    "\n",
    "maximize\n",
    "\n",
    "## 第149页第7行\n",
    "\n",
    "如果它们满足$\\mathbf{p}_i\\mathbf{F}\\mathbf{p}_j=\\mathbf{0}$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "如果它们满足$\\mathbf{p}_i^\\text{T}\\mathbf{F}\\mathbf{p}_j=0$\n",
    "\n",
    "\n",
    "## 第150页第5行\n",
    "\n",
    "$\\frac{\\partial}{\\partial\\alpha_k}\\left(\\frac{1}{2}{{\\left(\\mathbf{x}_k+\\alpha_k\\mathbf{p}_k\\right)}^\\mathrm{T}}\\mathbf{F}\\left(\\mathbf{x}_k+\\alpha_k\\mathbf{p}_k\\right)-\\mathbf{g}^{\\mathrm{T}}\\left(\\mathbf{x}_k+\\alpha\\mathbf{p}_k\\right)\\right)=\\alpha_k\\mathbf{p}_k^\\mathrm{T}\\mathbf{F}\\mathbf{p}_k-\\mathbf{p}_k^\\mathrm{T}\\left(\\mathbf{F}\\mathbf{x}_k-\\mathbf{g}\\right)$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\frac{\\partial}{\\partial\\alpha_k}\\left(\\frac{1}{2}{{\\left(\\mathbf{x}_k+\\alpha_k\\mathbf{p}_k\\right)}^\\mathrm{T}}\\mathbf{F}\\left(\\mathbf{x}_k+\\alpha_k\\mathbf{p}_k\\right)-\\mathbf{g}^{\\mathrm{T}}\\left(\\mathbf{x}_k+\\alpha_k\\mathbf{p}_k\\right)\\right)=\\alpha_k\\mathbf{p}_k^\\mathrm{T}\\mathbf{F}\\mathbf{p}_k+\\mathbf{p}_k^\\mathrm{T}\\left(\\mathbf{F}\\mathbf{x}_k-\\mathbf{g}\\right)$\n",
    "\n",
    "\n",
    "\n",
    "## 第150页第7行\n",
    "\n",
    "$\\alpha_k=\\frac{\\mathbf{p}_k^\\mathrm{T}\\left(\\mathbf{F}\\mathbf{x}_k-\\mathbf{g}\\right)}{\\mathbf{p}_k^\\mathrm{T}\\mathbf{F}\\mathbf{p}_k}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\alpha_k=\\frac{\\mathbf{p}_k^\\mathrm{T}\\left(\\mathbf{g}-\\mathbf{F}\\mathbf{x}_k\\right)}{\\mathbf{p}_k^\\mathrm{T}\\mathbf{F}\\mathbf{p}_k}$\n",
    "\n",
    "\n",
    "## 第151页第6行\n",
    "\n",
    "minimize\n",
    "\n",
    "### 改为\n",
    "\n",
    "maximize\n",
    "\n",
    "## 第154页算法8-10第2.2步\n",
    "\n",
    "2.2（决定初始动作）用$b\\left(\\cdot|\\sfS\\right)$得到动作$\\sfA$；\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.2（决定初始状态动作对）选择状态$\\sfS$，用$b\\left(\\cdot|\\sfS\\right)$得到动作$\\sfA$；\n",
    "\n",
    "## 第154页算法8-10第2.3.2步\n",
    "\n",
    "得到动作$A'$\n",
    "\n",
    "### 改为\n",
    "\n",
    "得到动作$\\sfA'$\n",
    "\n",
    "## 第155页第27行\n",
    "\n",
    "$=-\\sum\\limits_\\sfa{\\frac{\\pi\\left(\\sfa|{\\sfS_t};\\bftheta^\\mathrm{EMA}\\right)}{\\pi\\left(\\sfa|{\\sfS_t};\\bftheta\\right)}}.$\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "$=-\\sum\\limits_\\sfa\\frac{\\pi\\left(\\sfa\\mid\\sfS_t;\\bftheta^\\mathrm{EMA}\\right)}{\\pi\\left(\\sfa\\mid\\sfS_t;\\bftheta\\right)} {\\nabla_{\\bftheta}} \\pi\\left(\\sfa\\mid\\sfS_t;\\bftheta\\right).$\n",
    "\n",
    "## 第156页第14~16行\n",
    "\n",
    "将前式代入后式可得$\\lambda_t=\\frac{\\mathbf{k}_t^\\textrm{T}\\mathbf{g}-\\delta}{\\mathbf{k}_t^\\textrm{T}\\mathbf{k}_t}$。由于Lagrange乘子应当大等于0，所以Lagrange乘子应\n",
    "\n",
    "为$\\max\\left\\{\\frac{\\mathbf{k}_t^\\textrm{T}\\mathbf{g}-\\delta}{\\mathbf{k}_t^\\textrm{T}\\mathbf{k}_t},0\\right\\}$，优化问题的最优解为\n",
    "\n",
    "$\\mathbf{z}_t=\\mathbf{g}_t-\\max\\left\\{\\frac{bf{k}_t^\\textrm{T}\\mathbf{g}-\\delta}{\\mathbf{k}_t^\\textrm{T}\\mathbf{k}_t},0\\right\\}\\mathbf{k}_t$\n",
    "\n",
    "### 改为\n",
    "\n",
    "将前式代入后式可得$\\lambda_t=\\frac{\\mathbf{k}_t^\\mathrm{T}\\mathbf{g}_t-\\delta}{\\mathbf{k}_t^\\mathrm{T}\\mathbf{k}_t}$。由于Lagrange乘子应当大等于0，所以Lagrange乘子应\n",
    "\n",
    "为$\\max\\left\\{\\frac{{\\mathbf{k}_t^\\textrm{T}\\mathbf{g}_t-\\delta}}{\\mathbf{k}_t^\\textrm{T}\\mathbf{k}_t},0\\right\\}$，优化问题的最优解为\n",
    "\n",
    "$\\mathbf{z}_t=\\mathbf{g}_t-\\max\\left\\{\\frac{\\mathbf{k}_t^\\mathrm{T}\\mathbf{g}_t-\\delta}{\\mathbf{k}_t^\\textrm{T}\\mathbf{k}_t},0\\right\\}\\mathbf{k}_t$\n",
    "\n",
    "## 第157页算法8-11第3.2.3步\n",
    "\n",
    "$\\mathbf{k}=\\nabla_{\\bftheta}{d_\\textrm{KL}}\\left(\\pi\\left(\\cdot\\mid\\sfS_t;\\bftheta^\\mathrm{EMA}\\right)||\\pi\\left(\\cdot\\mid\\sfS_t;\\bftheta\\right)\\right)$，$\\mathbf{z}_t=\\mathbf{g}_t-\\max\\left\\{\\frac{\\mathbf{k}_t^{\\rm{T}}\\mathbf{g}-\\delta}{\\mathbf{k}_t^\\rm{T}\\mathbf{k}_t},0\\right\\}\\mathbf{k}_t$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\mathbf{k}\\leftarrow{\\nabla_{\\bftheta}}{d_{{\\textrm{KL}}}}\\left(\\pi\\left(\\cdot|\\sfS_t;\\bftheta^\\mathrm{EMA}\\right)||\\pi\\left(\\cdot\\mid\\sfS_t;\\bftheta'\\right)\\right)$，$\\mathbf{z}\\leftarrow\\mathbf{g}-\\max\\left\\{\\frac{\\mathbf{k}^\\textrm{T}\\mathbf{g}-\\delta}{\\mathbf{k}^\\textrm{T}\\mathbf{k}},0\\right\\}\\mathbf{k}$\n",
    "\n",
    "\n",
    "## 第157页算法8-11\n",
    "\n",
    "将第3.3步重新编号为第3.2.4步。\n",
    "\n",
    "## 第158页第19行，第159页第1~2行\n",
    "\n",
    "$\\sfA_t=a$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\sfA_t=\\sfa$\n",
    "\n",
    "## 第159页第8行\n",
    "\n",
    "和最优动作价值函数。\n",
    "\n",
    "### 改为\n",
    "\n",
    "和最优状态价值函数。\n",
    "\n",
    "## 第159页第27行\n",
    "\n",
    "其中目标$U^{\\left(q\\right)}_t=R_{t+1}+\\gamma{v}\\left(\\sfS;\\mathbf{w}^\\left(v\\right)_\\textrm{target}\\right)$；\n",
    "\n",
    "### 改为\n",
    "\n",
    "其中目标$U^{\\left(q\\right)}_t=R_{t+1}+\\gamma{v}\\left(\\sfS';\\mathbf{w}^\\left(v\\right)_\\textrm{target}\\right)$；\n",
    "\n",
    "## 第160页算法8-12第2.2.2步\n",
    "\n",
    "对应的回报$U^{\\left(q\\right)}_t=R_{t+1}+\\gamma{v}\\left(\\sfS;\\mathbf{w}^\\left(v\\right)_\\textrm{target}\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "对应的回报$U^{\\left(q\\right)}_t=R_{t+1}+\\gamma{v}\\left(\\sfS';\\mathbf{w}^\\left(v\\right)_\\textrm{target}\\right)$\n",
    "\n",
    "## 第162页第7行\n",
    "\n",
    "$D''=\\frac{1}{2}\\cos\\Theta''+\\frac{5}{4}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$D''_t=\\frac{1}{2}\\cos\\Theta''_t+\\frac{5}{4}$\n",
    "\n",
    "## 第162页第11~12行\n",
    "\n",
    "使得角速度有界$\\dot\\Theta_1\\in\\left[-4\\pi,4\\pi\\right]$，$\\dot\\Theta_2\\in\\left[-9\\pi,9\\pi\\right]$。\n",
    "\n",
    "### 改为\n",
    "\n",
    "使得角速度有界$\\dot\\Theta'_t\\in\\left[-4\\pi,4\\pi\\right]$，$\\dot\\Theta''_t\\in\\left[-9\\pi,9\\pi\\right]$。\n",
    "\n",
    "\n",
    "## 第162页第17行\n",
    "\n",
    "单5-7中的`qlearning()`函数\n",
    "\n",
    "### 改为\n",
    "\n",
    "单5-7中的`play_sarsa()`函数\n",
    "\n",
    "## 第163页代码清单8-1\n",
    "\n",
    "其中的`learn()`函数代码改为\n",
    "\n",
    "```\n",
    "def learn(self, observation, action, reward, next_observation, done,\n",
    "        next_action=None):\n",
    "    # 训练执行者网络\n",
    "    x = observation[np.newaxis]\n",
    "    u = self.critic_net.predict(x)\n",
    "    q = u[0, action]\n",
    "    x_tensor = tf.convert_to_tensor(x, dtype=tf.float32)\n",
    "    with tf.GradientTape() as tape:\n",
    "        pi_tensor = self.actor_net(x_tensor)[0, action]\n",
    "        logpi_tensor = tf.math.log(tf.clip_by_value(pi_tensor,\n",
    "                1e-6, 1.))\n",
    "        loss_tensor = -self.discount * q * logpi_tensor\n",
    "    grad_tensors = tape.gradient(loss_tensor, self.actor_net.variables)\n",
    "    self.actor_net.optimizer.apply_gradients(zip(\n",
    "            grad_tensors, self.actor_net.variables))\n",
    "    \n",
    "    # 训练评论者网络\n",
    "    u[0, action] = reward\n",
    "    if not done:\n",
    "        q = self.critic_net.predict(\n",
    "                next_observation[np.newaxis])[0, next_action]\n",
    "        u[0, action] += self.gamma * q\n",
    "    self.critic_net.fit(x, u, verbose=0)\n",
    "    \n",
    "    if done:\n",
    "        self.discount = 1.\n",
    "    else:\n",
    "        self.discount *= self.gamma\n",
    "```\n",
    "\n",
    "## 第164页第1行\n",
    "\n",
    "中的`qlearning()`函数\n",
    "\n",
    "### 改为\n",
    "\n",
    "中的`play_qlearning()`函数\n",
    "\n",
    "## 第164页代码清单8-2\n",
    "\n",
    "其中的`learn()`函数代码改为\n",
    "\n",
    "```\n",
    "def learn(self, observation, action, reward, next_observation, done):\n",
    "    x = observation[np.newaxis] # 特征\n",
    "    u = reward + (1. - done) * self.gamma * self.critic_net.predict(\n",
    "            next_observation[np.newaxis]) # 评论者目标\n",
    "    td_error = u - self.critic_net.predict(x)\n",
    "    \n",
    "    # 训练执行者网络\n",
    "    x_tensor = tf.convert_to_tensor(observation[np.newaxis],\n",
    "            dtype=tf.float32)\n",
    "    with tf.GradientTape() as tape:\n",
    "        pi_tensor = self.actor_net(x_tensor)[0, action]\n",
    "        logpi_tensor = tf.math.log(tf.clip_by_value(pi_tensor, 1e-6, 1.))\n",
    "        loss_tensor = -self.discount * td_error * logpi_tensor\n",
    "    grad_tensors = tape.gradient(loss_tensor, self.actor_net.variables)\n",
    "    self.actor_net.optimizer.apply_gradients(zip(\n",
    "            grad_tensors, self.actor_net.variables)) # 更新执行者网络\n",
    "    \n",
    "    # 训练评论者网络\n",
    "    self.critic_net.fit(x, u, verbose=0) # 更新评论者网络\n",
    "    \n",
    "    if done:\n",
    "        self.discount = 1. # 为下一回合初始化累计折扣\n",
    "    else:\n",
    "        self.discount *= self.gamma # 进一步累计折扣\n",
    "```\n",
    "\n",
    "## 第168页第3行\n",
    "\n",
    "的`qlearning()`函数联合起来\n",
    "\n",
    "### 改为\n",
    "\n",
    "的`play_qlearning()`函数联合起来\n",
    "\n",
    "## 第172页第17行\n",
    "\n",
    "$q_{\\pi\\left(\\bftheta\\right)}\\left(\\sfs,\\pi\\left(\\sfs;\\bftheta\\right)\\right)=r\\left(\\sfs,\\pi\\left(\\sfs;\\bftheta\\right)\\right)+\\gamma\\sum\\limits_{s'}{p\\left(\\sfs'|\\sfs,\\pi\\left(\\bftheta\\right)\\right){v_{\\pi\\left(\\bftheta\\right)}}\\left(\\sfs'\\right)},\\quad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$q_{\\pi\\left(\\bftheta\\right)}\\left(\\sfs,\\pi\\left(\\sfs;\\bftheta\\right)\\right)=r\\left(\\sfs,\\pi\\left(\\sfs;\\bftheta\\right)\\right)+\\gamma\\sum\\limits_{\\sfs'}{p\\left(\\sfs'|\\sfs,\\pi\\left(\\sfs;\\bftheta\\right)\\right){v_{\\pi\\left(\\bftheta\\right)}}\\left(\\sfs'\\right)},\\quad\\sfs\\in\\mathcal{S}$\n",
    "\n",
    "## 第173页第2~4行、第6行、第9行（共5处）\n",
    "\n",
    "$\\sum\\limits_{\\sfs'}{p\\left(\\sfs'|\\sfs;\\pi\\left(\\bftheta\\right)\\right)\\nabla{v_{\\pi\\left(\\bftheta\\right)}}\\left(\\sfs'\\right)}$\n",
    "\n",
    "\n",
    "改为\n",
    "\n",
    "$\\sum\\limits_{\\sfs'}{p\\left(\\sfs'|\\sfs,\\pi\\left(\\sfs;\\bftheta\\right)\\right)\\nabla{v_{\\pi\\left(\\bftheta\\right)}}\\left(\\sfs'\\right)}$\n",
    "\n",
    "\n",
    "## 第173页第20行\n",
    "\n",
    "$=\\E\\left[\\nabla\\pi\\left(\\sfS_0;\\bftheta\\right){\\left[\\nabla_\\sfa q_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_0,\\sfa\\right)\\right]}_{\\sfa=\\pi\\left(\\sfS_0;\\bftheta\\right)}\\right]+\\gamma\\E\\left[\\nabla\\pi\\left(\\sfS_1;\\bftheta\\right){{\\left[{\\nabla_\\sfa}{q_{\\pi\\left(\\bftheta\\right)}}\\left(\\sfS_1,\\sfa\\right)\\right]}_{\\sfa=\\pi\\left(\\sfS_1;\\bftheta\\right)}}\\right]+\\gamma^2\\E\\left[\\nabla{v}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_1\\right)\\right]$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$=\\E\\left[\\nabla\\pi\\left(\\sfS_0;\\bftheta\\right){\\left[\\nabla_\\sfa q_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_0,\\sfa\\right)\\right]}_{\\sfa=\\pi\\left(\\sfS_0;\\bftheta\\right)}\\right]+\\gamma\\E\\left[\\nabla\\pi\\left(\\sfS_1;\\bftheta\\right){{\\left[{\\nabla_\\sfa}{q_{\\pi\\left(\\bftheta\\right)}}\\left(\\sfS_1,\\sfa\\right)\\right]}_{\\sfa=\\pi\\left(\\sfS_1;\\bftheta\\right)}}\\right]+\\gamma^2\\E\\left[\\nabla{v}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS_2\\right)\\right]$\n",
    "\n",
    "\n",
    "## 第175页第7行\n",
    "\n",
    "其中$\\sfA_\\min$和$\\sfA_\\max$是动作的最小取值和最大取值\n",
    "\n",
    "### 改为\n",
    "\n",
    "其中$\\sfA_\\text{low}$和$\\sfA_\\text{high}$是动作的最小取值和最大取值\n",
    "\n",
    "## 第175页算法9-1第2.2步\n",
    "\n",
    "2.2（决定初始动作）\n",
    "\n",
    "### 改为\n",
    "\n",
    "2.2（初始化状态动作对）选择状态$\\sfS$，\n",
    "\n",
    "## 第176页第8行\n",
    "\n",
    "方差为$\\frac{\\sigma^2}{2\\theta}\\left(1-e^{-2\\theta}\\right)$，\n",
    "\n",
    "### 改为\n",
    "\n",
    "方差为$\\frac{\\sigma^2}{2\\theta}\\left(1-e^{-2\\theta{t}}\\right)$，\n",
    "\n",
    "## 第176页到第177页9.2节正文\n",
    "\n",
    "### 改为\n",
    "\n",
    "对于连续的动作空间，我们希望能够找到一个确定性策略，使得每条轨迹的回报最大。同策确定性算法利用策略$\\pi\\left(\\bftheta\\right)$生成轨迹，并在这些轨迹上求得回报的平均值，通过让平均回报最大，使得每条轨迹上的回报尽可能大。事实上，如果每条轨迹的回报都要最大，那么对于任意策略$b$采样得到的轨迹，我们都希望在这套轨迹上的平均回报最大。所以异策确定性策略算法引入确定性行为策略$b$，将这个平均改为针对策略$b$采样得到的轨迹，得到异策确定性梯度为$\\nabla\\E_{\\rho_b}\\left[q_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS,\\pi\\left(\\sfS;\\bftheta\\right)\\right)\\right]=\\E_{\\rho_b}\\left[\\nabla\\pi\\left(\\sfS;\\bftheta\\right)\\left[\\nabla_a{q}_{\\pi\\left(\\bftheta\\right)}\\left(\\sfS,\\sfa\\right)\\right]_{\\sfa=\\pi\\left(\\sfS;\\bftheta\\right)}\\right]$。这个表达式与同策的情形相比，期望运算针对的表达式相同。所以，异策确定性算法的迭代式与同策确定性算法的迭代式相同。\n",
    "\n",
    "异策确定性算法可能比同策确定性算法性能好的原因在于，行为策略可能会促进探索，用行为策略采样得到的轨迹能够更加全面的探索轨迹空间。这时候，最大化对轨迹分布的平均期望时能够同时考虑到更多不同的轨迹，使得在整个轨迹空间上的所有轨迹的回报会更大。\n",
    "\n",
    "与非确定性策略梯度相比，非确定性异策算法的迭代式中含有重采样因子$\\frac{\\pi\\left(\\sfA_t|\\sfS_t;\\bftheta\\right)}{b\\left(\\sfA_t|\\sfS_t\\right)}$，而确定性异策算法中没有。这是因为，确定性的行为策略$b$并不对于确定性的目标策略$\\pi\\left(\\bftheta\\right)$绝对连续，重采样因子没有定义，所以不包括重采样因子。\n",
    "\n",
    "\n",
    "## 第177页算法9-2第2.1步和2.2步之间\n",
    "\n",
    "### 增加一步（新2.2步）\n",
    "\n",
    "2.2（初始化状态）选择状态$\\sfS$；\n",
    "\n",
    "## 第178页算法9-3第2.2.2步\n",
    "\n",
    "$U\\leftarrow{R}+\\gamma{q}\\left(\\sfS',\\pi\\left(\\sfS';\\bftheta\\right)\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U\\leftarrow{R}+\\gamma{q}\\left(\\sfS',\\pi\\left(\\sfS';\\bftheta_\\text{target}\\right);\\mathbf{w}_\\text{target}\\right)$\n",
    "\n",
    "\n",
    "## 第179页算法9-4第2.2.3步\n",
    "\n",
    "$U=R+\\gamma\\min_{i=0,1}q\\left(\\sfS',\\pi\\left(\\sfS';\\bftheta_\\text{target}\\right);\\mathbf{w}^{\\left(i\\right)}\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$U\\leftarrow{R}+\\gamma\\min_{i=0,1}q\\left(\\sfS',\\sfA';\\mathbf{w}^{\\left(i\\right)}_\\text{target}\\right)$\n",
    "\n",
    "\n",
    "## 第180页图9-1\n",
    "\n",
    "### 改为\n",
    "\n",
    "<img src=\"./images/figure09_01.png\" style=\"width: 200px;\"/>\n",
    "\n",
    "\n",
    "## 第180页第4行\n",
    "\n",
    "$X$轴是水平向下的，$Y$轴是水平向右的。\n",
    "\n",
    "### 改为\n",
    "\n",
    "$X$轴是水平向上的，$Y$轴是水平向左的。\n",
    "\n",
    "\n",
    "## 第180页第8行\n",
    "\n",
    "下一观测$\\left(\\cos\\Theta_{t+1},\\sin\\Theta_{t+1},\\dot\\Theta\\right)$。\n",
    "\n",
    "### 改为\n",
    "\n",
    "下一观测$\\left(\\cos\\Theta_{t+1},\\sin\\Theta_{t+1},\\dot\\Theta_{t+1}\\right)$。\n",
    "\n",
    "\n",
    "## 第180页表9-1最右边一列\n",
    "\n",
    "| 倒立摆（Pendulum-v0）|\n",
    "|----------|\n",
    "| $\\left[-\\pi,\\pi\\right)\\times\\left[-4\\pi,4\\pi\\right]\\times\\left[-9\\pi,9\\pi\\right]$ |\n",
    "| ${\\left[-1,1\\right]}^2\\times\\left[-4\\pi,4\\pi\\right]\\times\\left[-9\\pi,9\\pi\\right]$ |\n",
    "| $\\left[-2,2\\right]$ |\n",
    "| $\\left[-\\pi^2-6.404,0\\right]$ |\n",
    "\n",
    "\n",
    "### 改为\n",
    "\n",
    "| 倒立摆（Pendulum-v0）|\n",
    "|----------|\n",
    "| $\\left[-\\pi,\\pi\\right)\\times\\left[-8,8\\right]$ |\n",
    "| $\\left[-1,1\\right]^2\\times\\left[-8,8\\right]$ |\n",
    "| $\\left[-2,2\\right]$ |\n",
    "| $\\left[-\\pi^2-6.404,0\\right]$ |\n",
    "\n",
    "\n",
    "## 第180页第18-19行\n",
    "\n",
    "$\\Theta_{t+1}\\leftarrow\\Theta_t+0.05\\left(\\dot\\Theta-0.75\\sin\\Theta-0.15\\sfA_t\\right)$在$\\left[-\\pi,+\\pi\\right)$的主值区间\n",
    "\n",
    "$\\dot\\Theta_{t+1}\\leftarrow\\text{clip}\\left(\\dot\\Theta_t-0.75\\sin\\Theta_t-0.15\\sfA_t,-8,+8\\right)$\n",
    "\n",
    "### 改为\n",
    "\n",
    "$\\Theta_{t+1}\\leftarrow\\Theta_t+0.05\\left(\\dot\\Theta_t+0.75\\sin\\Theta_t+0.15\\sfA_t\\right)$在$\\left[-\\pi,+\\pi\\right)$的主值区间\n",
    "\n",
    "$\\dot\\Theta_{t+1}\\leftarrow\\text{clip}\\left(\\dot\\Theta_t+0.75\\sin\\Theta_t+0.15\\sfA_t,-8,+8\\right)$\n",
    "\n",
    "\n",
    "## 第181页第1行\n",
    "\n",
    "值$\\left|\\dot{\\Theta}\\right|$和\n",
    "\n",
    "### 改为\n",
    "\n",
    "值$\\left|\\dot{\\Theta}_t\\right|$和\n",
    "\n",
    "\n",
    "## 第189页\n",
    "\n",
    "`gym[atari]`已官方支持Windows。在Windows、macOS、Linux系统下均只需用下列命令就可以完成`gym[atari]`的安装：\n",
    "\n",
    "```\n",
    "pip install --upgrade gym[atari]\n",
    "```\n",
    "\n",
    "\n",
    "## 第204页第5~6行\n",
    "\n",
    "极大极小算法（monimax）\n",
    "\n",
    "### 改为\n",
    "\n",
    "极大极小算法（minimax）\n",
    "\n",
    "\n",
    "## 第207页图11-3\n",
    "\n",
    "### 改为\n",
    "\n",
    "<img src=\"./images/figure11_03.png\" style=\"width: 400px;\"/>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
